Disable CPU Cores in Linux

On May 27, 2015, in Uncategorized, by Jin

This does not require OS reboot.

  • Before the change:
    # cat /sys/devices/system/cpu/cpu3/online
    1
    # cat /proc/cpuinfo | grep processor | wc -l
    24
  • The change
    # echo 0 > /sys/devices/system/cpu/cpu3/online
  • After the change:
    # cat /sys/devices/system/cpu/cpu3/online
    0
    # cat /proc/cpuinfo | grep processor | wc -l
    23

To make a permanent change, add maxcpus=n to /etc/grub.conf

Tagged with:  

Improve scp/rsync performance

On April 28, 2015, in Uncategorized, by Jin

Before:
$ scp hostA:/tmp/6GB.gz /tmp
6GB.gz 100% 6118MB 58.3MB/s 01:45

Use the encryption with less resources and no compression:
$ scp -c arcfour -o Compression=no -o LogLevel=quiet hostA:/tmp/6GB.gz /tmp
6GB.gz 100% 6118MB 111.2MB/s 00:55

For rsync:
rsync -e ‘ssh -c arcfour -o Compression=no -o LogLevel=quiet -T -x’

It is also possible to set as default in ~user/.ssh/ssh_config:
Host *
Compression no
LogLevel quiet
Ciphers arcfour256,arcfour128,arcfour

Tagged with:  
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
#!/bin/bash
 
##################################
# Build Latest SaltStack RPM
##################################
 
# Setup environment variables
export http_proxy=http://USER:PASS@PROXY:PORT
export https_proxy=$http_proxy
TMP_DIR=/v1/tmp/salt_build.$$
PYTHON_VERSION=2.7.8 # Python 2.7.9 issue: http://stackoverflow.com/questions/28296476/python-pip-install-requires-server-hostname
PYTHON=Python-$PYTHON_VERSION
SALT=salt-2014.7.2
SETUPTOOLS=
INSTALL_DIR=/v1/apps/salt
INSTALL_BIN_DIR=$INSTALL_DIR/bin
OS=`cat /etc/redhat-release | cut -d' ' -f3`
 
echo Start building $SALT using $PYTHON to $INSTALL_DIR...
sleep 3
 
# Setup working directory
echo Making temporary work directory at $TMP_DIR
mkdir $TMP_DIR && cd $TMP_DIR
if [ $? -ne 0 ]
then
  exit Failed to create $TMP_DIR
  exit 1
else
  # Clean up previous build
  rm -rf $INSTALL_DIR
fi
 
# Download Packages
# Python
wget -m -nd https://www.python.org/ftp/python/$PYTHON_VERSION/$PYTHON.tgz
if [ $? -ne 0 ]
then
  echo Failed to download $PYTHON.
  exit 2
fi
# Salt
wget -m -nd --no-check-certificate https://pypi.python.org/packages/source/s/salt/$SALT.tar.gz
if [ $? -ne 0 ]
then
  echo Failed to download $SALT
  exit 3
fi
 
# Compile and build
# Python
tar xvf $PYTHON.tgz
cd $PYTHON
./configure --prefix=$INSTALL_DIR/python
make -j4
make install
cd $TMP_DIR
 
# Setup pip
wget --no-check-certificate https://bootstrap.pypa.io/get-pip.py
$INSTALL_DIR/python/bin/python get-pip.py
 
# Setup Salt
tar xvf $SALT.tar.gz
cd $SALT
 
if [ "$OS" == "5.6" ]
then
  $INSTALL_DIR/python/bin/pip install Cython
  $INSTALL_DIR/python/bin/pip install https://github.com/Bluehorn/pyzmq/archive/0b7769addc4ff1da77f453bf1273a71cb74d8844.zip
fi
$INSTALL_DIR/python/bin/pip install -r zeromq-requirements.txt
$INSTALL_DIR/python/bin/python setup.py install
cd ..
 
# Create symbolic links
echo Creating symbolic links under $INSTALL_BIN_DIR
mkdir $INSTALL_BIN_DIR && cd $INSTALL_BIN_DIR
if [ $? -ne 0 ]
then
  echo Failed to create $INSTALL_BIN_DIR
fi
for f in `ls ../python/bin/salt*`
do
  ln -s $f
done
 
# Done.
echo Done.
 
 

Upgrade opengrok

On September 12, 2014, in Uncategorized, by Jin

1. Download opengrok binary package and extract it to the destination folder with version number


# ls /apps/opengrok*
/apps/opengrok-0.11.1
/apps/opengrok-0.12.1

2. Stop opengrok application in Tomcat manager

3. Deploy new opengrok

export OPENGROK_TOMCAT_BASE=/apps/tomcat
export JAVA_HOME=/opt/jdk1.7.0_51
/apps/opengrok-0.12.1/bin/OpenGrok deploy

4. Move previous etc and data folder if no need to reindex.

# mv /apps/opengrok-0.11.1/data /apps/opengrok-0.12.1
# mv /apps/opengrok-0.11.1/etc /apps/opengrok-0.12.1

5. Modify /apps/opengrok-0.12.1/etc/configuration.xml with new path name

6. Start the application in Tomcat manager

Tagged with:  

SaltStack Python renderer examples

On June 18, 2014, in Uncategorized, by Jin

Reference:
salt.renderers.py

#!py
# mystate.sls
def run():
  config = {}
  config ['/tmp/%s' % __pillar__['linkname'] ] = {
    'file.symlink': [
      { 'target' : __grains__['host'], 
        'user'   : 'userA',
        'group'  : 'groupA' } ] }
  return config

Call it to create /tmp/hostfile symlink pointing to /tmp/`hostname`

salt 'minion.domain.com' state.sls mystate pillar="{'linkname':'hostfile'}"

Tagged with:  

SaltStack In A Large Deployment

On June 17, 2014, in Uncategorized, by Jin

On Master:
* A disk with fast IO
* worker_thread: increase worker threads to handle more return calls made from minions
* keep_jobs: Decrease the number of hours to keep old job information
* timeout: increase default timeout for the salt command and API
* loop_interval: increase master maintenance process check cycle to run backend jobs less frequently
* job_cache: Disable job cache to speed up the server process

On Minions:
* random_reauth_delay: Increase to avoid SYN flood to the server.
* auth_timeout: Increase waiting timeout for a master to accept minion’s public key
* recon_default/recon_max: increase default/maximum wait time before trying to reconnect to master
* recon_randomize: Enable to avoid flooding the server

Tagged with:  

YUM tips

On June 10, 2014, in Uncategorized, by Jin

# List all repos
# yum repolist

# Install from selected repos if RPMs with dependencies came from different repos.
# yum –disablerepo=’*’ –enablerepo=’repo1,repo2′ install glibc.i686 nss-softokn-freebl

Tagged with:  

Get User SMTP addresses from AD

On March 7, 2014, in Uncategorized, by Jin

This code needs to be improved to check subgroups.

Import-Module activedirectory
$users = get-adgroupmember 'WINDOWS_GROUP_NAME'
ForEach ($u in $users)
{
write-host -NoNewline $u.SamAccountName
get-aduser $u.SamAccountName -Properties proxyaddresses | %{$_.proxyaddresses} | ?{$_.ToLower().startswith("smtp")} | write-host -NoNewline
write-host " "
}

Tagged with:  

Multicast test samples

On March 4, 2014, in Uncategorized, by Jin

Sender:
msend -b 1000 -n 3 -m 64 -s 100 -q 239.239.239.100 1000 5 interface_ip
iperf -c 239.0.1.229 -u -T 32 -t 3 -i 1

Receiver:
mdump -s -q -v 239.239.239.100 interface_ip
iperf -s -u -B 239.0.1.229 -i 1

Tagged with: