FPGA Boot

With the eventual goal of booting uCLinux on SCOORE, the work is currently focusing on booting uCLinux on OpenSPARC T1. This project can be nicely divided into several high-level steps:

• Synthesize T1 core
• Place and route design
• Operating system & software
• Load onto board

This project entailed the exploration of each of these sections, but is currently incomplete. We were never able to get the Xilinx EDK to place and route OpenSPARC T1 1.6 and ran out of time.

Synthesize T1 Core

The first step is downloading the T1 core which can be done here (registration required). Simply unzip it, setup environment variables, and synthesize.

Prerequisites

A 32-bit installation of Linux is the easiest, as there're some issues with 64-bit Linux and one of the programs, though you can get it to work. Any distro, in our case Centos 5, should work. Custom software such as synplicity_pro, synplify, VCS, or VERA may be needed depending on if you're doing testing or synthesis.

Environment Variables

I used Bash for this project, and the bash script provided is actually tcsh. This script should be run on login or before doing synthesis. After converting it and deciding what I needed I ended up with the following:

export DV_ROOT="~/OpenSPARC_T1";
export MODEL_DIR="~/OpenSPARC_T1_model";
export CC_BIN="/usr/bin";
export SYNP_HOME="${SOFT}/synplicity/fpga";
export PATH=".:$DV_ROOT/tools/bin:$CC_BIN/:$PATH";
export ENVDIR="$DV_ROOT/tools/env";

export TRE_ENTRY="/";
export TRE_LOG="nobody";
export TRE_SEARCH="$ENVDIR/tools.iver";

export PERL_MODULE_BASE="$DV_ROOT/tools/perlmod";
export PERL_CMD="/usr/bin/perl";
export VERA_HOME=""

# Important for VCS to not throw a warning
export VCS_ARCH_OVERRIDE="linux"

# Synthesis and simulation commands
export SIMS="sims -vcs_build_args=-full64 -group=core1 -novera_run -novera_build -nosas";
export SYN="rsynp -device=XC5VLX110 -all";

The last two aliases, SIMS and SYN, are for simulation of the core and synthesis respectively. They're for a system without SAS or VERA on 64-bit Linux compiling for the XC5VLX110 (university edition/ML509).

Running on 64-bit

64-bit Linux requires a little more customization than when using 32-bit.

Configure /tools/bin/syn_command to add a -64bit argument. The resulting line should look something like: dc_shell-t -64bit -f $DV_ROOT/design/sys/synopsys/script/run.scr > dc_shell.log

The sims command should be passed a vcs_build_args argument of -full64. The code I'm using to run the regression suite for OpenSPARC is sims -vcs_build_args=-full64 -group=core1 -novera_run -novera_build.

The configsrch tool within DV_ROOT/tools/Linux/x86_64 failed out with an unhelpful error code. I believe mktools was run at some point and that helped, but in the end it was replaced with the following shell script (may only apply to the 1.6 T1 release):

#!/bin/bash
case "$1" in
  "sims" ) echo "1.262";;
  "DiagList" ) echo "1.11";;
  "OutputDirector" ) echo "1.01";;
  "Sims" ) echo "1.12";;
  "showargv" ) echo "1.01";;
  "regreport" ) echo "1.73";;
  "Regreport" ) echo "5.01";;
  "BitFieldTie" ) echo "1.09";;
  "Midas" ) echo "3.30";;
  * ) echo "1.0";;
esac

Further editing of some C/C++ files is required to get this running fully: (First two items from the Sun forums, last comment points 2 & 4)

1. Files $DV_ROOT/tools/pli/iop/device.h and $DV_ROOT/tools/pli/iop/pcx.h have an extra qualification, namely void pcx::clean(); in pcx.h needed to be changed to void clean(); and a similar change needed to be made in device.h. I'm fairly certain that this won't break functionality with any older compilers and will help to fix it for newer ones.
2. $DV_ROOT/verif/env/iss/pli/common/c/CircularLinkedList.cc failed to compile because line 174 was head = null instead of head = NULL.
3. Edited device.h:44 and removed device::

Synthesis

There are a few synthesis configuration options available in $DV_ROOT/design/sys/synplicity/env.prj. The following options were used by us.

• FPGA_SYN - Resizes various megacells and arrays to line up better in FPGAs and removes all asyncronous logic.
• FPGA_SYN_NO_SPU - Removes the Stream Processing Unit for cryptographic acceleration.
• FPGA_SYN_8_TLB - Decreases the TLB from 64 to 16 entries. (FPGA_SYN_16_TLB also exists)
• FPGA_SYN_1THREAD - Reduces the single core from 4-threads to 1-thread capabilities.

Synthesis of the Sparc core and extra modules such as the crossbar bus is done using either the rsyn or rsynp programs. In the Bash script above we created as alias for synthesis that is SYN, so to start synthesis we just ran that.

We were originally only interested in synthesis and never made it past that, so we ended up with EDIF netlists of several different components that can be found in DV_ROOT/design/sys/iop/MODULE/SOFTWARE/HARDWARE. For example after running rsynp -device=XC5VLX110 sparc you will find the output in DV_ROOT/design/sys/iop/sparc/synplicity/XC5VLX110. sparc.edf is the important file that is used with the EDK project.

Software

uClinux development first starts with a Linux distribution running on top of a SPARC computer. Ubuntu 7.10 can be found here: [1]. The Sun server in the MASC lab is being setup precisely for this purpose.

This part of the project concerned the creation of a Linux disk image that could be booted from. It would need to be able to integrated into the workflow and combined with the hardware part to be placed onto the FPGA. I have been in communication with Gopal Reddy form Sun as they are working on a Linux port to the OpenSPARC T1 also. He explained the procedure as being 1) generating an empty disk image, 2) copying a bare 7.10 SPARC Ubuntu into that disk image, and 3) adding extra kernel modules and drivers for the FPGA.

Gopal included some binaries and they are available from Professor Renau.

Provided Ramdisks

OpenSPARC T1 1.6 comes with a couple of RAMdisks that are explained in $DV_ROOT\edk\os\bin\docs\t1_fpga_opensolaris_faq.txt.

1. DV_ROOT\edk\os\bin\proto\ubuntu-gutsy-linux-ramdisk.80mb.gz
2. DV_ROOT\edk\os\OpenSolaris\proto\ramdisk.snv-b77-nd.gz (not sure about this one)

Inspection of a ramdisk is done as follows:

# losetup /dev/loop0 ubuntu-gutsy-linux-ramdisk.80mb
# mount /dev/loop0 /mnt/ramdisk
# ls /mnt/ramdisk
-- DISPLAYS CONTENTS --

Create FPGA ACE Image

To generate an ACE file from a hardware bitfile, ramdisk, and software on Windows execute the following in an EDK shell: xmd -tcl genace.tcl -jprog -target mdm -hw bitfiles/download_4t_16tlb_lx110t.bit -data os/bin/proto/ubuntu-gutsy-linux-ramdisk.80mb.gz 0x5af00000 -board ml505 -ace PWNZ.ace -data os/bin/proto/1c1t_obp_prom.bin 0x5ff00000 -alf ccx-firmware/executable.elf A similar procedure should work in Linux.

Info can be found here: Using the GenAce Script.

For information on memory map, see ccx-firmware\src\mbfw_config.h.

Booting the FPGA

Using the Xilinx EDK method of placing code on the board currently fails. Two forum threads have been started here and here with the Sun one possibly having the solution.

From ACE File

• Load the appropriate .ACE file onto the CompactFlash card. The .ACE file for a full OpenSolaris boot can be found in $SPARC/design/sys/edk/ace/OpenSPARCT1_1_6_os_boot.ace
• The ML505 board boots from one of the folders on the CompactFlash card under the config directory. Pick a folder rev[0-7] and set the CFG[2:0] bits of SW3 to the binary value of the chosen folder. Be sure that there is only one .ACE file per directory.
• Currently, rev0 contains the 'Hello World' image, while rev1 contains the full OpenSolaris boot image.
• Attach a serial cable to the computer and connect to the serial connection using 9600 baud, 8N1, no-parity. Putty's default options work fine on Windows or cu
• When the board is powered up, it will eventually prompt you with {OK}. at this point you can choose one of a couple boot options: boot -mverbose: 60 minute boot up time, initializes all services, including telnet and networking or boot -m milestone=none: faster boot, no services.

Using the Xilinx EDK

The Xilinx EDK provides the fastest way to compile and load new hardware or software onto the board.

1. A default SPARC netlist is provided with OpenSPARC, but if you have generated your own copy it from ~/OpenSPARC_T1/design/sys/iop/sparc/synplicity/XC5VLX110 to ~/OpenSPARC_T1/design/sys/edk/pcores/iop_fpga_v1_00_a/netlist.
2. Change permissions for the edk project by running chmod -R u+w edk or updating the project to 10.1 will fail among other things
3. Launch the Xilinx EDK from the command line by running xps.
4. Open the project ~/OpenSPARC_T1/design/sys/edk/system.xmp.
5. Select Hardware→Generate Bitstream from the menu to generate and load the bitstream for the FPGA.

References

OpenSPARC on FPGA Project Site