Difference between revisions of "Desktop Setup - Ubuntu 12.04"

From Vlsiwiki
Jump to: navigation, search
(MADA Related Stuff)
(MADA Related Stuff)
 
(45 intermediate revisions by 9 users not shown)
Line 48: Line 48:
  
 
* 3 - Setup the new repo
 
* 3 - Setup the new repo
  scp <username>@mascd12:/etc/apt/sources.list /etc/apt/source.list
+
  scp <username>@mada0:/etc/apt/sources.list /etc/apt/sources.list
 
  apt-get update -y
 
  apt-get update -y
 
  apt-get dist-upgrade -y
 
  apt-get dist-upgrade -y
Line 62: Line 62:
 
  echo -e "\nfs.inotify.max_user_watches = 65535 \nkernel.randomize_va_space = 0" | tee -a /etc/sysctl.conf
 
  echo -e "\nfs.inotify.max_user_watches = 65535 \nkernel.randomize_va_space = 0" | tee -a /etc/sysctl.conf
  
* 5 - Replicate the installation on Master machine (mascd12, dhcp-63-154.cse.ucsc.edu)
+
* 5 - Replicate the installation on Master machine (mascd3)
 
The current apt-get/dpkg releases have problem. So we need to work around the problems that is going to happen in this process.
 
The current apt-get/dpkg releases have problem. So we need to work around the problems that is going to happen in this process.
 +
Make sure the machine you are copying from is an Ubuntu 12.04 machine.
  
  ssh <username>@mascd12 dpkg --get-selections > dpkg.list
+
  ssh <username>@mascd3 dpkg --get-selections > dpkg.list.full
  dpkg --set-selections < dpkg.list
+
grep -v i386 dpkg.list.full >dpkg.list.x64
 +
grep i386 dpkg.list.full >dpkg.list.i386
 +
 
 +
  dpkg --set-selections < dpkg.list.x64
 
  apt-get update -y
 
  apt-get update -y
 +
 +
dpkg --set-selections < dpkg.list.i386
 +
apt-get update -y
 +
 +
then remove the problematic packages from the package selecting list, and run the apt-get dselect-upgrade again :
  
 
  echo "libdb5.1-java:i386    deinstall" | dpkg --set-selections
 
  echo "libdb5.1-java:i386    deinstall" | dpkg --set-selections
Line 80: Line 89:
 
It most likely will fail, complaining unable to install libapt-pkg4.12:i386, etc. You might need to repeat this until the apt-get dselect-upgrade finished gracefully. Pay attention to the error message. If it asks you to do 'apt-get install -f', then do so!
 
It most likely will fail, complaining unable to install libapt-pkg4.12:i386, etc. You might need to repeat this until the apt-get dselect-upgrade finished gracefully. Pay attention to the error message. If it asks you to do 'apt-get install -f', then do so!
  
finalized the installed packages so far:
+
Finalized the installed packages so far:
  
 
  dpkg --configure -a
 
  dpkg --configure -a
  
then remove the problematic packages from the package selecting list, and run the apt-get dselect-upgrade again :
+
==Ruby Related Stuff==
  
echo "libdb5.1-java:i386    deinstall" | dpkg --set-selections
+
We use RVM to support multiple ruby versions
echo "libsensors4:i386    deinstall" | dpkg --set-selections
+
 
echo "libapt-pkg4.12:i386    deinstall" | dpkg --set-selection
+
* Install the base versions shared by all the users
  apt-get dselect-upgrade -y
+
 
 +
  sudo apt-get install curl
 +
 
 +
Log as user (foo) and install the latest rvm
 +
 
 +
* Do '''not use sudo''' when installing RVM
 +
 
 +
* RVM only needs to be installed once per use
 +
 
 +
  curl -L https://get.rvm.io | sudo bash -s stable
 +
 
 +
Add yourself (foo) to to the rvm group (vi /etc/group)
 +
 
 +
Install the following ruby options
 +
 
 +
  rvm install ruby-1.8.7
 +
  rvm install rbx
 +
  rvm install rbx-2.0.0-rc1 # This may be better in the future, but rbx 2 has several advantages
 +
  rvm install ruby-1.9.3
 +
  rvm install jruby
 +
 
 +
* Install the packages needed. These should work across ruby versions:
 +
 
 +
rvm use 1.9.3
 +
gem install rspec parslet treetop trollop rgl json gherkin cucumber
 +
 
 +
rvm use 1.8.7
 +
rvm rubygems 1.3.5
 +
gem install rspec-expectations
 +
  gem install rspec parslet treetop trollop rgl json gherkin cucumber
 +
gem install reek flay flog
 +
 
 +
rvm use rbx
 +
gem install rspec parslet treetop trollop rgl json gherkin cucumber
 +
 
 +
... # any ruby version that you may want to use
 +
 
 +
* This are version specific
 +
 
 +
rvm use 1.8.7
 +
gem install
 +
 +
rvm use 1.9.3
 +
gem install
 +
 
 +
rvm use rbx-2.0.0-rc1
 +
gem install
  
 
==MADA Related Stuff==
 
==MADA Related Stuff==
Line 99: Line 154:
 
Get autofs files from another machine
 
Get autofs files from another machine
 
  apt-get install -y autofs  
 
  apt-get install -y autofs  
  scp <username>@mascd12:/etc/auto.* /etc/
+
  scp <username>@mada7:/etc/auto.* /etc/
 
  ln -s /mada/users /soe
 
  ln -s /mada/users /soe
 
  initctl reload autofs
 
  initctl reload autofs
 +
 +
/etc/auto.mada should use nfs4. Something like this:
 +
users              -rw,intr,soft,noquota,noatime,fstype=nfs4 mada0.cse.ucsc.edu:/mada/users
 +
software            -ro,intr,soft,noquota,noatime,fstype=nfs4 mada0.cse.ucsc.edu:/mada/software
 +
 +
edit /etc/default/nfs-common. If you have a NEED_IDMAPD variable, it must be set to true
 +
 +
NEED_IDMAPD=yes
  
 
* 7 - awk for cadence
 
* 7 - awk for cadence
Line 112: Line 175:
 
  ufw reject auth
 
  ufw reject auth
 
  ufw limit ssh/tcp
 
  ufw limit ssh/tcp
 +
ufw logging off
  
* 9 - Get tempcap from another ubuntu machine (mascd1, mascd12, ...), and some lib preparation:
+
* 9 - Get tempcap from another ubuntu machine (mascd3, mascd9, ...), and some lib preparation:
 
   
 
   
mkdir /usr/lib64
+
  scp <username>@mada0:/usr/lib/libtermcap.so.2.0.8 /usr/lib/
  scp <username>@mascd12:/usr/lib64/libtermcap.so.2.0.8 /usr/lib64/
+
  ln -s /usr/lib/libtermcap.so.2.0.8 /usr/lib/libtermcap.so.2
  ln -s /usr/lib64/libtermcap.so.2.0.8 /usr/lib64/libtermcap.so.2
+
  
  ln -s /usr/lib/libtiff.so.4 /usr/lib/libtiff.so.3
+
  ln -sf /usr/lib/x86_64-linux-gnu/libtiff.so.4.3.4 /usr/lib/libtiff.so.3
  
 
* 10 - LDAP
 
* 10 - LDAP
Line 128: Line 191:
  
 
Create /ect/krb5.conf
 
Create /ect/krb5.conf
  scp <username>@mascd1:/etc/krb5.conf /etc/
+
  scp <username>@mascd12:/etc/krb5.conf /etc/
  
 
Copy the ldap certificate
 
Copy the ldap certificate
  scp <username>@mascd1:/etc/ssl/certs/* /etc/ssl/certs/
+
  scp <username>@mascd12:/etc/ssl/certs/* /etc/ssl/certs/
 +
scp -r <username>@mada4:/etc/openldap/cacerts /etc/openldap/.
  
 
Create "/etc/auth-client-config/profile.d/krb-ldap-config"
 
Create "/etc/auth-client-config/profile.d/krb-ldap-config"
  scp <username>@mascd1:/etc/auth-client-config/profile.d/krb-ldap-config /etc/auth-client-config/profile.d/krb-ldap-config
+
  scp <username>@mascd12:/etc/auth-client-config/profile.d/krb-ldap-config /etc/auth-client-config/profile.d/krb-ldap-config
 
  auth-client-config -a -p krb_ldap
 
  auth-client-config -a -p krb_ldap
  
 
Update ldap.conf
 
Update ldap.conf
  scp <username>@mascd1:/etc/ldap.conf /etc/ldap.conf
+
  scp <username>@mascd12:/etc/pam_ldap.conf /etc/ldap.conf
  scp <username>@mascd1:/etc/ldap/ldap.conf /etc/ldap/ldap.conf
+
  scp <username>@mada7:/etc/ldap/ldap.conf /etc/ldap/ldap.conf
  
 
Now you should be able to execute:
 
Now you should be able to execute:
Line 145: Line 209:
  
 
* 11 - Lock SOE people out of your machine :)
 
* 11 - Lock SOE people out of your machine :)
  scp <username>@mascd12:/etc/security/access.conf /etc/security/access.conf
+
  scp <username>@mada7:/etc/security/access.conf /etc/security/access.conf
  
 
You will need to add your local login to the machine, or you will lock yourself out:
 
You will need to add your local login to the machine, or you will lock yourself out:
Line 154: Line 218:
 
  + : root : ALL
 
  + : root : ALL
 
  + : nx  : ALL
 
  + : nx  : ALL
+ : ian_local : ALL
 
 
  + : (masc) : ALL
 
  + : (masc) : ALL
 
  + : (vlsi) : ALL
 
  + : (vlsi) : ALL
Line 160: Line 223:
  
 
Now, you should see people with "getent passwd" but, calling "su - cormac" should say "permission denied", but, calling "su - renau" should work.
 
Now, you should see people with "getent passwd" but, calling "su - cormac" should say "permission denied", but, calling "su - renau" should work.
 +
If it doesn't work, run
 +
 +
getent passwd renau
 +
getent passwd | grep renau
 +
 +
If the result is empty for the first line, but not empty for the second one, restart nscd:
 +
 +
service nscd restart
  
 
* 12 - Since the LDAP has timeouts, setup the local backup
 
* 12 - Since the LDAP has timeouts, setup the local backup
Line 166: Line 237:
 
  nss_updatedb ldap
 
  nss_updatedb ldap
 
  cd /etc/cron.hourly/
 
  cd /etc/cron.hourly/
  scp <username>@mada1:/etc/cron.hourly/nss_updatedb_ldap .
+
  scp <username>@mada7:/etc/cron.hourly/nss_updatedb_ldap .
  scp <username>@mada1:/etc/nsswitch.conf /etc
+
  scp <username>@mada7:/etc/nsswitch.conf /etc
  
 
* 13 - Edit lines passwd and group in /etc/nsswitch.conf so that they are as follows (involves adding db before ldap)  
 
* 13 - Edit lines passwd and group in /etc/nsswitch.conf so that they are as follows (involves adding db before ldap)  
  
 
   sudo vi /etc/nsswitch.conf
 
   sudo vi /etc/nsswitch.conf
   passwd:        files db ldap  
+
   passwd:        files db [NOTFOUND=return] ldap  
   group:          files db ldap
+
   group:          files db [NOTFOUND=return] ldap
  
 
* 14 - Allow other accounts to be shown at the login screen, and disable Guest user.
 
* 14 - Allow other accounts to be shown at the login screen, and disable Guest user.
Line 184: Line 255:
 
  allow-guest=false
 
  allow-guest=false
  
 +
* 16 - Edit the /etc/idmapd.conf and make sure that you use the correct idmapd for nfs4. It should
 +
set:
 +
 +
Domain = localdomain
  
 
* 15 - Restart your computer:
 
* 15 - Restart your computer:
Line 209: Line 284:
  
 
* 20 - Using QEMU and ARM in 32 bit mode. We need to install 32 bit libraries for this (as root).  
 
* 20 - Using QEMU and ARM in 32 bit mode. We need to install 32 bit libraries for this (as root).  
  sudo apt-get install g++-multilib lib32z1 lib32z1-dev lib32ncurses5 lib32ncurses5-dev ia32-libs libxext-dev:i386 -y
+
  sudo apt-get install g++-multilib libxext-dev:i386 -y
  
* 21 - Ruby Gems
 
  
  sudo apt-get install ruby rubygems
+
* 21 - Install some fonts
  sudo gem install --remote builder colored cucumber diff-lcs gherkim nanoc2 polyglot rake rgl rspec stream treetop
+
  sudo gem install cri --version 1.0.1
+
  
* 22 - Install some fonts
+
  sudo apt-get install libtiff4 t1-xfree86-nonfree ttf-xfree86-nonfree ttf-xfree86-nonfree-syriac xfonts-75dpi xfonts-100dpi
  
  sudo apt-get install t1-xfree86-nonfree ttf-xfree86-nonfree ttf-xfree86-nonfree-syriac xfonts-75dpi xfonts-100dpi
 
  
 
+
* 22 - Make sure that there is no file indexing (beagle)
* 23 - Make sure that there is no file indexing (beagle)
+
  
 
  sudo apt-get purge beagle
 
  sudo apt-get purge beagle
  
* 24 - Add to /etc/crontab to maintain same time (13:13 use any other random time)
+
* 23 - Add/create to /etc/cron.daily/ntpdate for keep time adjusted (chmod 755 ntpdate)
  
  echo -e "13 13  * * *  root    /usr/sbin/ntpdate ntp.ucsc.edu" | sudo tee -a /etc/crontab
+
  #!/bin/sh
 +
ntpdate-debian
  
* 25 - Add a freenx server to your desktop
+
* 24 - Add a freenx server to your desktop
  
 
  sudo apt-get install python-software-properties
 
  sudo apt-get install python-software-properties
Line 237: Line 308:
 
  sudo apt-get install freenx-server
 
  sudo apt-get install freenx-server
  
* 26 - Add sendmail (no external forwarding)
+
* 25 - Add sendmail (no external forwarding)
  
 
  sudo apt-get purge exim4
 
  sudo apt-get purge exim4
Line 249: Line 320:
  
  
* 27- Install extra tools (should have been installed through dpkg --set-selection)
+
* 26- Install extra tools (should have been installed through dpkg --set-selection)
  
   sudo apt-get install -y git-core
+
   sudo apt-get install -y git-core libjpeg62
   sudo apt-get install -y libglib2.0-dev
+
   sudo apt-get install -y libglib2.0-dev zlib1g-dev build-essential
 
   sudo apt-get install -y bison flex
 
   sudo apt-get install -y bison flex
 
   sudo apt-get install -y libboost-dev
 
   sudo apt-get install -y libboost-dev
Line 262: Line 333:
 
   sudo apt-get install -y okular
 
   sudo apt-get install -y okular
  
* 28 - Copy libtermcap needed by some synopsys aps
+
* 27 - Copy libtermcap needed by some synopsys aps
  
 
   sudo scp <username>@mada0:/usr/lib/libtermcap.so.2.0.8 /usr/lib/.
 
   sudo scp <username>@mada0:/usr/lib/libtermcap.so.2.0.8 /usr/lib/.
   sudo ln -s libtermcap.so.2.0.8 libtermcap.so.2
+
   sudo ln -s /usr/lib/libtermcap.so.2.0.8 /usr/lib/libtermcap.so.2
  
* 29 - Manually install glib for i386, and get some more 32 bit libs.
+
* 28 - Manually install glib for i386, and get some more 32 bit libs.
  
 +
  scp mascd9:/lib/i386-linux-gnu/libg.tar /tmp/
 
   cd /lib/i386-linux-gnu
 
   cd /lib/i386-linux-gnu
   sudo tar zxvf /mada/software/libg.tar
+
   sudo tar zxvf /tmp/libg.tar
  
 
==Upgrade from Ubuntu 10.04==
 
==Upgrade from Ubuntu 10.04==
Line 278: Line 350:
 
   https://help.ubuntu.com/community/MaverickUpgrades
 
   https://help.ubuntu.com/community/MaverickUpgrades
  
==Installing grid-engine (server)==
+
==Installing PBS Torque (server)==
 +
 
 +
* mada0 will be the headnode (pbsserver). Install the following packages on mada0:
 +
 
 +
apt-get install torque-server torque-scheduler torque-client
 +
 
 +
* on other mada machines
 +
 
 +
apt-get install torque-client torque-mom
 +
 
 +
 
 +
* Torque will use TCP ports 15000-15004. Either open the ports or allow them for mada machines based on their IP address.
 +
 
 +
ufw allow proto tcp from EACH_MADA_IP to any port 15000:15004
 +
 
 +
* Make sure server_name is properly set
 +
 
 +
vi /var/spool/torque/server_name
 +
 
 +
mada0
 +
 
 +
 
 +
* on mada0, add the other mada machines as compute node:
 +
 
 +
vi /var/spool/torque/server_priv/nodes
 +
 
 +
* and add the following
 +
 
 +
mada8 np=16
 +
mada1 np=48
 +
mada2 np=4
 +
mada3 np=4
 +
mada4 np=4
 +
mada5 np=4
 +
mada6 np=4
 +
mada7 np=4
 +
mada0 np=10
 +
 
 +
on the all mada machines, set the pbsserver
 +
 
 +
vi /var/spool/torque/mom_priv/config
 +
 
 +
* add the following lines:
 +
$pbsserver mada0
 +
$logevent  255
 +
 
 +
* Now create a queue and run the server on mada0
 +
 
 +
qmgr -c "set server scheduling=true"
 +
qmgr -c "create queue batch queue_type=execution"
 +
qmgr -c "set queue batch started=true"
 +
qmgr -c "set queue batch enabled=true"
 +
qmgr -c "set queue batch resources_default.walltime=240:00:00"
 +
qmgr -c "set server default_queue=batch"
 +
qmgr -c "set server operators = root@mada0"
 +
qmgr -c "set server keep_completed = 0"
 +
qmgr -c "set queue batch resources_default.neednodes = 1:ppn=1"
 +
 
 +
* restart the pbs_server, pbs_client and pbs_sched on mada0, and pbs_mom and pbs_client on others
 +
 
 +
* pbsnode -a should show the status of all nodes. If nodes are down, check the firewall log to make sure it does not block the requests.
 +
 
 +
* check the queue
 +
echo "sleep 10" | qsub -l nodes=1:ppn=1
 +
qstat
 +
 
 +
==Configuring mdadm RAID (servers)==
 +
 
 +
* For mada0, we use mdadm raid 5.
 +
 
 +
* To assemble the hard drives in a raid array:
 +
 
 +
mdadm --assemble /dev/md0 /dev/disk/by-id/ID-OF-DISK /dev/disk/by-id/ID-OF-DISK2 ...
 +
 
 +
You can preserve the same alphabetic order in which the list of disk IDs appear.
 +
 
 +
Note that this assembles the raid /dev/md0 from the devices already prepared for raid.
 +
You might need to stop the raid if it is already running to be able to reassemble it:
 +
 
 +
mdadm --stop /dev/md0
 +
 
 +
* If the disk have not been prepared for raid before, then you need to create the raid partition tables first. We use the whole disk for raids.
 +
 
 +
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=6 /dev/disk/by-id/ID-OF-DISK1 /dev/disk/by-id/ID-OF-DISK2 ...
 +
 
 +
You can preserve the same alphabetic order in which the list of disk IDs appear.
 +
 
 +
To see the preparation that will happen in the background:
 +
 
 +
watch cat /proc/mdstat
 +
 
 +
To see the details of the array:
 +
 
 +
mdadm --detail /dev/md0
 +
 
 +
* Then you need to format the disks:
 +
 
 +
mkfs.ext4 /dev/md0
 +
 
 +
* update the /etc/fstab.
 +
 
 +
Lets say we want to mount the raid at /mirror
 +
 
 +
mkdir /mirror
 +
vi /etc/fstab
 +
 
 +
and make sure you specify the mount point for the raind.
 +
 
 +
/dev/md0        /mirror            ext4        rw,intr,soft,noquota,relatime    0        0
 +
 
 +
then mount the raid:
 +
 
 +
mount -a
 +
 
 +
==Configuring zfs RAID (servers)==
 +
 
 +
* Install the zfs kernel module (not fuse)
 +
 
 +
sudo add-apt-repository ppa:zfs-native/stable
 +
 
 +
* Format the SDDs to have the following partitions:
 +
1 /boot : 4GB
 +
2 cache : 80GB
 +
3 log: 4GB
 +
4 root  : 32GB
 +
5 swap : 4GB
 +
 
 +
* create the radz1 zpool
 +
 
 +
  zpool create tank raidz1 sdc sdd sde sdf sdg sdh
 +
  zpool status tank
 +
 
 +
  zpool iostat -v
 +
 
 +
* add read and write caches. To make sure you add the right partition, use "lsblk" to ensure the right partition.
 +
 +
  zppol add tank cache /dev/disk/by-id/ata-FOO_disk1_part2  /dev/disk/by-id/ata-FOO_disk2_part2 log mirror /dev/disk/by-id/ata-FOO_disk1_part3 /dev/disk/by-id/ata-FOO_disk2_part3
 +
 
 +
  zpool iostat -v
 +
 
 +
* Now prepare the folders
 +
 
 +
zfs create tank/users
 +
zfs create tank/software
 +
zfs create tank/servers
 +
 
 +
 
 +
==Setting up CUPS for Printing==
 +
 
 +
* 1. Make sure cups and cups-bsd are installed:
 +
 
 +
  sudo apt-get install cups
 +
 
 +
* 2. Go to /etc/cups directory:
 +
 
 +
  cd /etc/cups
 +
 
 +
* 3. Add file called "client.conf" with this in it (you may need sudo access to create new file):
 +
 
 +
  ServerName cups-01.cse.ucsc.edu
 +
  ServerName spooler4.cse.ucsc.edu
 +
  spooler4.cse.ucsc.edu
 +
 
 +
* 4. Save the client.conf file and restart the cups server:
 +
 
 +
  /etc/init.d/cups restart
 +
 
 +
* You should now be able to print.  If you are not seeing a list of printer or are getting an error saying something like "client error: forbidden", then your machine does not have access to the VLAN printer network.  Contact the IT tech staff and this issue should be resolved (Eric Shell helped with this).
 +
 
  
 +
==Setting up Cadence Virtuoso on your Ubuntu 12.04 Desktop==
  
* Firewall
+
1. '''cd /tmp'''
  
ufw allow in 6444
+
2. '''sudo apt-get update'''
ufw allow in 6445
+
  
* packages
+
3. '''sudo apt-get install nscd csh ksh gawk libelf1 libelf-dev ia32-libs libelf1:i386'''
  
apt-get purge gridengine-exec
+
This installs the library dependencies.
apt-get install gridengine-exec gridengine-client
+
  
* Set SGE master
+
4. '''sudo apt-get update'''
  
echo "mada0.cse.ucsc.edu" >/var/lib/gridengine/default/common/act_qmaster
+
5. '''sudo ln -s /usr/bin/awk /bin/awk'''
  
* Start daemon (do not wait for the next reboot)
+
This ensures the awk utility is also available at the locations typically found on Solaris, CentOS, SuSE flavors.
  
/etc/init.d/gridengine-exec restart
+
6. For Hspice simulation, you may need to install libtiff4
 +
  '''apt-get install libtiff4'''
 +
If it won't work due to version difference, create a soft link using following command
 +
  '''sudo ln -s /usr/lib/x86_64-linux-gnu/libtiff.so.4 /usr/lib/libtiff.so.3'''

Latest revision as of 20:21, 14 January 2015

Currently still testing the distribution, so please report any problems to Ehsan and Jose!!

  • 0 - Install Ubuntu 12.04 x86_64

NOTE: ONLY if you have trouble with Step 4 of the installation process, the partition selection, open a terminal in the trial mode and run:

sudo dmraid -r -E /dev/sda

NOTE: When prompted for a username, choose one that is NOT your SOE login! For example "ian_local" rather than my SOE login of "ianlee1521". Otherwise you will not be able to login to your computer after you setup LDAP.

NOTE: When prompted for a machine name, enter the name that machine is supposed to be (mascd1, omelette, quiche, etc)

Installation should complete as normal.



Plain Ubuntu Installation

  • 1 - Enable root account

Login to the GUI with your local login, then open a terminal (Applications -> Accessories -> Terminal):

sudo su - 
passwd

Enter your new root password for the machine.

Add your local and SOE logins to the sudoers list:

vi /etc/sudoers
  • 2 - Install NVIDIA Drivers

Blacklist some kernel modules

vi /etc/modprobe.d/blacklist.conf

Add the following lines to the file

blacklist vga16fb
blacklist nouveau
blacklist rivafb
blacklist nvidiafb
blacklist rivatv

Get rid of any installed NVIDIA drivers (you are already root, if not use with sudo):

apt-get --purge remove nvidia-* 
apt-get update
apt-get install nvidia-current


Reboot the machine and X should start normally. Login at the GUI login as root and open a terminal.

  • 3 - Setup the new repo
scp <username>@mada0:/etc/apt/sources.list /etc/apt/sources.list
apt-get update -y
apt-get dist-upgrade -y


  • 4 - Adjust sysctl options

mmap problem for qemu

vi /etc/sysctl.d/10-zeropage.conf

Edit the following value:

vm.mmap_min_addr = 4096

increase (add) the inotify watches in /etc/sysctl.conf

echo -e "\nfs.inotify.max_user_watches = 65535 \nkernel.randomize_va_space = 0" | tee -a /etc/sysctl.conf
  • 5 - Replicate the installation on Master machine (mascd3)

The current apt-get/dpkg releases have problem. So we need to work around the problems that is going to happen in this process. Make sure the machine you are copying from is an Ubuntu 12.04 machine.

ssh <username>@mascd3 dpkg --get-selections > dpkg.list.full
grep -v i386 dpkg.list.full >dpkg.list.x64
grep i386 dpkg.list.full >dpkg.list.i386
dpkg --set-selections < dpkg.list.x64
apt-get update -y
dpkg --set-selections < dpkg.list.i386
apt-get update -y

then remove the problematic packages from the package selecting list, and run the apt-get dselect-upgrade again :

echo "libdb5.1-java:i386    deinstall" | dpkg --set-selections
echo "libsensors4:i386    deinstall" | dpkg --set-selections
echo "libapt-pkg4.12:i386    deinstall" | dpkg --set-selections

"apt-get dselect-upgrade -y" Will give many options and take a long time for the upgrade (about an hour for me). Select all default options.

apt-get dselect-upgrade -y

It most likely will fail, complaining unable to install libapt-pkg4.12:i386, etc. You might need to repeat this until the apt-get dselect-upgrade finished gracefully. Pay attention to the error message. If it asks you to do 'apt-get install -f', then do so!

Finalized the installed packages so far:

dpkg --configure -a

Ruby Related Stuff

We use RVM to support multiple ruby versions

  • Install the base versions shared by all the users
 sudo apt-get install curl

Log as user (foo) and install the latest rvm

  • Do not use sudo when installing RVM
  • RVM only needs to be installed once per use
 curl -L https://get.rvm.io | sudo bash -s stable

Add yourself (foo) to to the rvm group (vi /etc/group)

Install the following ruby options

 rvm install ruby-1.8.7
 rvm install rbx
 rvm install rbx-2.0.0-rc1 # This may be better in the future, but rbx 2 has several advantages
 rvm install ruby-1.9.3
 rvm install jruby
  • Install the packages needed. These should work across ruby versions:
rvm use 1.9.3 
gem install rspec parslet treetop trollop rgl json gherkin cucumber
rvm use 1.8.7
rvm rubygems 1.3.5
gem install rspec-expectations
gem install rspec parslet treetop trollop rgl json gherkin cucumber
gem install reek flay flog
rvm use rbx
gem install rspec parslet treetop trollop rgl json gherkin cucumber
... # any ruby version that you may want to use
  • This are version specific
rvm use 1.8.7
gem install

rvm use 1.9.3
gem install
rvm use rbx-2.0.0-rc1
gem install

MADA Related Stuff

  • 6 - Install autofs

Create mada directory

mkdir /mada

Get autofs files from another machine

apt-get install -y autofs 
scp <username>@mada7:/etc/auto.* /etc/
ln -s /mada/users /soe
initctl reload autofs

/etc/auto.mada should use nfs4. Something like this:

users               -rw,intr,soft,noquota,noatime,fstype=nfs4 mada0.cse.ucsc.edu:/mada/users 
software            -ro,intr,soft,noquota,noatime,fstype=nfs4 mada0.cse.ucsc.edu:/mada/software 

edit /etc/default/nfs-common. If you have a NEED_IDMAPD variable, it must be set to true

NEED_IDMAPD=yes

  • 7 - awk for cadence
apt-get install -y ksh csh original-awk
ln -s /usr/bin/awk /bin/awk
  • 8 - Set the firewall
ufw enable
ufw default deny
ufw reject auth
ufw limit ssh/tcp
ufw logging off
  • 9 - Get tempcap from another ubuntu machine (mascd3, mascd9, ...), and some lib preparation:
scp <username>@mada0:/usr/lib/libtermcap.so.2.0.8 /usr/lib/
ln -s /usr/lib/libtermcap.so.2.0.8 /usr/lib/libtermcap.so.2
ln -sf /usr/lib/x86_64-linux-gnu/libtiff.so.4.3.4 /usr/lib/libtiff.so.3
  • 10 - LDAP

Install

apt-get install -y ldap-auth-client libpam-krb5 krb5-user libsasl2-modules-gssapi-mit

Leave the kerberos config entry blank. You will copy it in the next step.

Create /ect/krb5.conf

scp <username>@mascd12:/etc/krb5.conf /etc/

Copy the ldap certificate

scp <username>@mascd12:/etc/ssl/certs/* /etc/ssl/certs/
scp -r <username>@mada4:/etc/openldap/cacerts /etc/openldap/.

Create "/etc/auth-client-config/profile.d/krb-ldap-config"

scp <username>@mascd12:/etc/auth-client-config/profile.d/krb-ldap-config /etc/auth-client-config/profile.d/krb-ldap-config
auth-client-config -a -p krb_ldap

Update ldap.conf

scp <username>@mascd12:/etc/pam_ldap.conf /etc/ldap.conf
scp <username>@mada7:/etc/ldap/ldap.conf /etc/ldap/ldap.conf

Now you should be able to execute:

ldapsearch -x LLL
  • 11 - Lock SOE people out of your machine :)
scp <username>@mada7:/etc/security/access.conf /etc/security/access.conf

You will need to add your local login to the machine, or you will lock yourself out:

vim /etc/security/access.conf

Go to very bottom of file and add your local user account to the list below root Example of the last lines of my file (Replace "ian_local" with your local login):

+ : root : ALL
+ : nx   : ALL
+ : (masc) : ALL
+ : (vlsi) : ALL
-:ALL :ALL 

Now, you should see people with "getent passwd" but, calling "su - cormac" should say "permission denied", but, calling "su - renau" should work. If it doesn't work, run

getent passwd renau
getent passwd | grep renau

If the result is empty for the first line, but not empty for the second one, restart nscd:

service nscd restart
  • 12 - Since the LDAP has timeouts, setup the local backup
apt-get install nss-updatedb
nss_updatedb ldap
cd /etc/cron.hourly/
scp <username>@mada7:/etc/cron.hourly/nss_updatedb_ldap .
scp <username>@mada7:/etc/nsswitch.conf /etc
  • 13 - Edit lines passwd and group in /etc/nsswitch.conf so that they are as follows (involves adding db before ldap)
 sudo vi /etc/nsswitch.conf
 passwd:         files db [NOTFOUND=return] ldap 
 group:          files db [NOTFOUND=return] ldap
  • 14 - Allow other accounts to be shown at the login screen, and disable Guest user.
 sudo vi /etc/lightdm/lightdm.conf
 

add the following lines to lightdm.conf

greeter-show-manual-login=true
allow-guest=false
  • 16 - Edit the /etc/idmapd.conf and make sure that you use the correct idmapd for nfs4. It should

set:

Domain = localdomain
  • 15 - Restart your computer:
shutdown -r now
  • 16 - Login to the GUI with your SOE login information. If you get a prompt asking for an "LDAP Password" this means that you typed your password incorrectly.
  • 17 - Remove old desktop configuration files (KDE / GNOME / browser)

NOTE: Removing ".config" will delete any saved passwords in your browser, but will fix other problems. NOTE: You can use kubuntu(KDE), gnome, unity, or awesome as desktop/windows manager.

Logout of / kill any open X sessions. Switch to tty1 (CTRL + ALT + F1), and login with your SOE account.

rm -rf .config .kde* .gconf* .gnome* .awesome*
  • 18 - Some Minor Patches for things that don't work properly

Disable apparmor

sudo invoke-rc.d apparmor stop
sudo update-rc.d -f apparmor remove
  • 19 - Add bash as default shell
cd /bin
sudo ln -sf bash sh
  • 20 - Using QEMU and ARM in 32 bit mode. We need to install 32 bit libraries for this (as root).
sudo apt-get install g++-multilib libxext-dev:i386 -y


  • 21 - Install some fonts
 sudo apt-get install libtiff4 t1-xfree86-nonfree ttf-xfree86-nonfree ttf-xfree86-nonfree-syriac xfonts-75dpi xfonts-100dpi


  • 22 - Make sure that there is no file indexing (beagle)
sudo apt-get purge beagle
  • 23 - Add/create to /etc/cron.daily/ntpdate for keep time adjusted (chmod 755 ntpdate)
#!/bin/sh
ntpdate-debian
  • 24 - Add a freenx server to your desktop
sudo apt-get install python-software-properties
sudo add-apt-repository ppa:freenx-team
sudo apt-get update
sudo apt-get install freenx-server
  • 25 - Add sendmail (no external forwarding)
sudo apt-get purge exim4
sudo apt-get install sendmail
cd /etc/mail/
sudo scp <username>@mada0:/etc/mail/alias* .
sudo scp <username>@mada0:/etc/mail/sendmail.* .
sudo scp <username>@mada0:/etc/mail/generi* .
sudo chmod 777 /var/spool/mqueue
sudo /etc/init.d/sendmail restart


  • 26- Install extra tools (should have been installed through dpkg --set-selection)
 sudo apt-get install -y git-core libjpeg62 
 sudo apt-get install -y libglib2.0-dev zlib1g-dev build-essential
 sudo apt-get install -y bison flex
 sudo apt-get install -y libboost-dev
 sudo apt-get install -y ack-grep
 sudo apt-get install -y cmake
 sudo apt-get install -y gridengine-client libtool mutrace
 sudo apt-get install -y texlive-latex-base texlive-latex-extra texlive-latex-recommended preview-latex-style latex-xcolor latex-beamer texlive-science texlive-fonts-recommended
 sudo apt-get install -y okular
  • 27 - Copy libtermcap needed by some synopsys aps
 sudo scp <username>@mada0:/usr/lib/libtermcap.so.2.0.8 /usr/lib/.
 sudo ln -s /usr/lib/libtermcap.so.2.0.8 /usr/lib/libtermcap.so.2
  • 28 - Manually install glib for i386, and get some more 32 bit libs.
 scp mascd9:/lib/i386-linux-gnu/libg.tar /tmp/
 cd /lib/i386-linux-gnu
 sudo tar zxvf /tmp/libg.tar

Upgrade from Ubuntu 10.04

  • 1- sudo apt-get dist-upgrade -y
  • 2- Upgrade to Ubuntu 1.10:
 https://help.ubuntu.com/community/MaverickUpgrades

Installing PBS Torque (server)

  • mada0 will be the headnode (pbsserver). Install the following packages on mada0:
apt-get install torque-server torque-scheduler torque-client
  • on other mada machines
apt-get install torque-client torque-mom


  • Torque will use TCP ports 15000-15004. Either open the ports or allow them for mada machines based on their IP address.
ufw allow proto tcp from EACH_MADA_IP to any port 15000:15004
  • Make sure server_name is properly set
vi /var/spool/torque/server_name
mada0


  • on mada0, add the other mada machines as compute node:
vi /var/spool/torque/server_priv/nodes
  • and add the following
mada8 np=16
mada1 np=48
mada2 np=4
mada3 np=4
mada4 np=4
mada5 np=4
mada6 np=4
mada7 np=4
mada0 np=10

on the all mada machines, set the pbsserver

vi /var/spool/torque/mom_priv/config
  • add the following lines:
$pbsserver mada0
$logevent  255
  • Now create a queue and run the server on mada0
qmgr -c "set server scheduling=true"
qmgr -c "create queue batch queue_type=execution"
qmgr -c "set queue batch started=true"
qmgr -c "set queue batch enabled=true"
qmgr -c "set queue batch resources_default.walltime=240:00:00"
qmgr -c "set server default_queue=batch"
qmgr -c "set server operators = root@mada0"
qmgr -c "set server keep_completed = 0"
qmgr -c "set queue batch resources_default.neednodes = 1:ppn=1"
  • restart the pbs_server, pbs_client and pbs_sched on mada0, and pbs_mom and pbs_client on others
  • pbsnode -a should show the status of all nodes. If nodes are down, check the firewall log to make sure it does not block the requests.
  • check the queue
echo "sleep 10" | qsub -l nodes=1:ppn=1
qstat

Configuring mdadm RAID (servers)

  • For mada0, we use mdadm raid 5.
  • To assemble the hard drives in a raid array:
mdadm --assemble /dev/md0 /dev/disk/by-id/ID-OF-DISK /dev/disk/by-id/ID-OF-DISK2 ...

You can preserve the same alphabetic order in which the list of disk IDs appear.

Note that this assembles the raid /dev/md0 from the devices already prepared for raid. You might need to stop the raid if it is already running to be able to reassemble it:

mdadm --stop /dev/md0
  • If the disk have not been prepared for raid before, then you need to create the raid partition tables first. We use the whole disk for raids.
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=6 /dev/disk/by-id/ID-OF-DISK1 /dev/disk/by-id/ID-OF-DISK2 ...

You can preserve the same alphabetic order in which the list of disk IDs appear.

To see the preparation that will happen in the background:

watch cat /proc/mdstat

To see the details of the array:

mdadm --detail /dev/md0
  • Then you need to format the disks:
mkfs.ext4 /dev/md0
  • update the /etc/fstab.

Lets say we want to mount the raid at /mirror

mkdir /mirror
vi /etc/fstab

and make sure you specify the mount point for the raind.

/dev/md0        /mirror            ext4        rw,intr,soft,noquota,relatime     0        0

then mount the raid:

mount -a

Configuring zfs RAID (servers)

  • Install the zfs kernel module (not fuse)
sudo add-apt-repository ppa:zfs-native/stable
  • Format the SDDs to have the following partitions:

1 /boot : 4GB 2 cache : 80GB 3 log: 4GB 4 root  : 32GB 5 swap : 4GB

  • create the radz1 zpool
 zpool create tank raidz1 sdc sdd sde sdf sdg sdh
 zpool status tank
 zpool iostat -v
  • add read and write caches. To make sure you add the right partition, use "lsblk" to ensure the right partition.
 zppol add tank cache /dev/disk/by-id/ata-FOO_disk1_part2  /dev/disk/by-id/ata-FOO_disk2_part2 log mirror /dev/disk/by-id/ata-FOO_disk1_part3 /dev/disk/by-id/ata-FOO_disk2_part3
 zpool iostat -v
  • Now prepare the folders

zfs create tank/users zfs create tank/software zfs create tank/servers


Setting up CUPS for Printing

  • 1. Make sure cups and cups-bsd are installed:
 sudo apt-get install cups
  • 2. Go to /etc/cups directory:
 cd /etc/cups
  • 3. Add file called "client.conf" with this in it (you may need sudo access to create new file):
 ServerName cups-01.cse.ucsc.edu
 ServerName spooler4.cse.ucsc.edu
 spooler4.cse.ucsc.edu
  • 4. Save the client.conf file and restart the cups server:
 /etc/init.d/cups restart
  • You should now be able to print. If you are not seeing a list of printer or are getting an error saying something like "client error: forbidden", then your machine does not have access to the VLAN printer network. Contact the IT tech staff and this issue should be resolved (Eric Shell helped with this).


Setting up Cadence Virtuoso on your Ubuntu 12.04 Desktop

1. cd /tmp

2. sudo apt-get update

3. sudo apt-get install nscd csh ksh gawk libelf1 libelf-dev ia32-libs libelf1:i386

This installs the library dependencies.

4. sudo apt-get update

5. sudo ln -s /usr/bin/awk /bin/awk

This ensures the awk utility is also available at the locations typically found on Solaris, CentOS, SuSE flavors.

6. For Hspice simulation, you may need to install libtiff4

 apt-get install libtiff4

If it won't work due to version difference, create a soft link using following command

 sudo ln -s /usr/lib/x86_64-linux-gnu/libtiff.so.4 /usr/lib/libtiff.so.3