in Openstack

Simulating DevStack jobs with Vagrant

There are times where you would like to troubleshoot locally why your change is failing a dsvm job at the gate.
Or maybe you want to write a new dsvm job definition and you need an environment as close
as the gate as possible to try out things.

The DevStack-gate project has instructions to simulate Devstack gate tests, which can be found here:

https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst

These instructions are easy to follow, but would be nice to have automation around it so I wrote a Vagrantfile:

if ENV['ZUUL_URL']
    ZUUL_URL = ENV['ZUUL_URL']
else
    ZUUL_URL = 'https://git.openstack.org'
end

if ENV['ZUUL_PROJECT']
    ZUUL_PROJECT = ENV['ZUUL_PROJECT']
else
    ZUUL_PROJECT = 'openstack/nova'
end

if ENV['ZUUL_BRANCH']
    ZUUL_BRANCH = ENV['ZUUL_BRANCH']
else
    ZUUL_BRANCH = 'master'
end

if ENV['ZUUL_REF']
    ZUUL_REF = ENV['ZUUL_REF']
else
    ZUUL_REF = 'HEAD'
end

if ENV['DEVSTACK_JOB']
    DEVSTACK_JOB = ENV['DEVSTACK_JOB']
else
    DEVSTACK_JOB = <<JOB
export PYTHONUNBUFFERED=true
export DEVSTACK_GATE_TEMPEST=1
export DEVSTACK_GATE_TEMPEST_FULL=1
cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
./safe-devstack-vm-gate-wrap.sh
JOB
end

$script = <<SCRIPT
apt-get update
apt-get -y install git python python-dev python-pip
pip install tox
ssh-keygen -N "" -t rsa -f /root/.ssh/id_rsa
KEY_CONTENTS=$(cat /root/.ssh/id_rsa.pub | awk '{print $2}' )
git clone https://git.openstack.org/openstack-infra/system-config /opt/system-config
/opt/system-config/install_puppet.sh
/opt/system-config/install_modules.sh
puppet apply --modulepath=/opt/system-config/modules:/etc/puppet/modules -e 'class { openstack_project::single_use_slave: install_users => false,   enable_unbound => true, ssh_key => \"$KEY_CONTENTS\" }'
echo "jenkins ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/jenkins
export WORKSPACE=/home/jenkins/workspace/testing
mkdir -p "$WORKSPACE"
cd $WORKSPACE
git clone --depth 1 https://git.openstack.org/openstack-infra/devstack-gate

export ZUUL_URL=#{ZUUL_URL}
export ZUUL_PROJECT=#{ZUUL_PROJECT}
export ZUUL_BRANCH=#{ZUUL_BRANCH}
export ZUUL_REF=#{ZUUL_REF}

exec 0<&-
#{DEVSTACK_JOB}
SCRIPT

Vagrant.configure(2) do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.provider "virtualbox" do |vb|
  #   # Display the VirtualBox GUI when booting the machine
  #   vb.gui = true
  #
  #   # Customize the amount of memory on the VM:
    vb.memory = "6144"
  end
  config.vm.provision "shell", inline: $script
end

This Vagrantfile has several variables that can be defined by the user:

ZUUL_URL -> The URL to clone the ref from
ZUUL_PROJECT -> The project to test
ZUUL_BRANCH -> The branch to test
ZUUL_REF -> The ref to test
DEVSTACK_JOB -> The dsvm job

The variables can be either modified on the Vagrantfile or defined as environment variables in the shell.

But better, let’s do an example:
I wrote the patch https://review.openstack.org/#/c/302770/ , which is about adding Designate zones support to the shade project.
This patch could use some functional/integration tests, but shade project jobs do not set up Designate.
Ideally, we would like to change one of the existing dsvm jobs for shade and just add the bits that make the produced DevStack “designate-enabled”.
So, we clone the project-config project, and write down one of the shade dsvm jobs:

export PYTHONUNBUFFERED=true
export DEVSTACK_GATE_NEUTRON=0
export DEVSTACK_GATE_HEAT=1
export DEVSTACK_LOCAL_CONFIG="enable_plugin designate git://git.openstack.org/openstack/designate"
export PROJECTS="openstack/designate $PROJECTS"
export PROJECTS="openstack/designate-dashboard $PROJECTS"
export PROJECTS="openstack/designate-tempest-plugin $PROJECTS"
export PROJECTS="openstack-infra/shade $PROJECTS"
if [ "$BRANCH_OVERRIDE" != "default" ] ; then
    export OVERRIDE_ZUUL_BRANCH=$BRANCH_OVERRIDE
fi

function post_test_hook {
    $BASE/new/shade/shade/tests/functional/hooks/post_test_hook.sh
}
export -f post_test_hook

cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
./safe-devstack-vm-gate-wrap.sh

By looking on the project-config Designate dsvm jobs, it seems that Designate is installed in DevStack via a plugin, so adding those lines to the shade dsvm jobs would do the job:

export PYTHONUNBUFFERED=true
export DEVSTACK_GATE_NEUTRON=0
export DEVSTACK_GATE_HEAT=1
export DEVSTACK_LOCAL_CONFIG="enable_plugin designate git://git.openstack.org/openstack/designate"
export PROJECTS="openstack/designate $PROJECTS"
export PROJECTS="openstack/designate-dashboard $PROJECTS"
export PROJECTS="openstack/designate-tempest-plugin $PROJECTS"
export PROJECTS="openstack-infra/shade $PROJECTS"
if [ "$BRANCH_OVERRIDE" != "default" ] ; then
    export OVERRIDE_ZUUL_BRANCH=$BRANCH_OVERRIDE
fi

function post_test_hook {
    $BASE/new/shade/shade/tests/functional/hooks/post_test_hook.sh
}
export -f post_test_hook

cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
./safe-devstack-vm-gate-wrap.sh

We just inserted lines 4 to 7, which enables the Designate DevStack plugin and adds the needed Designate projects to the PROJECTS variable, so they are checked out by DevStack-gate scripts.

We are almost there!
We just need to go to the change, click on ‘Download’ and annotate the ref to set the ZUUL_REF variable.
This is how you would run the whole thing:

ricky@ricky-Surface-Pro-3:~/devel/vagrants/devstack-designate$ export ZUUL_PROJECT=openstack-infra/shade
ricky@ricky-Surface-Pro-3:~/devel/vagrants/devstack-designate$ export ZUUL_BRANCH=master
ricky@ricky-Surface-Pro-3:~/devel/vagrants/devstack-designate$ export ZUUL_REF=refs/changes/70/302770/1
ricky@ricky-Surface-Pro-3:~/devel/vagrants/devstack-designate$ read -d '' DEVSTACK_JOB <<"EOF"
export PYTHONUNBUFFERED=true
export DEVSTACK_GATE_NEUTRON=0
export DEVSTACK_GATE_HEAT=1
export DEVSTACK_LOCAL_CONFIG="enable_plugin designate git://git.openstack.org/openstack/designate"
export PROJECTS="openstack/designate $PROJECTS"
export PROJECTS="openstack/designate-dashboard $PROJECTS"
export PROJECTS="openstack/designate-tempest-plugin $PROJECTS"
export PROJECTS="openstack-infra/shade $PROJECTS"
if [ "$BRANCH_OVERRIDE" != "default" ] ; then
    export OVERRIDE_ZUUL_BRANCH=$BRANCH_OVERRIDE
fi

function post_test_hook {
    $BASE/new/shade/shade/tests/functional/hooks/post_test_hook.sh
}
export -f post_test_hook

cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
./safe-devstack-vm-gate-wrap.sh
EOF
ricky@ricky-Surface-Pro-3:~/devel/vagrants/devstack-designate$ export DEVSTACK_JOB=$DEVSTACK_JOB
ricky@ricky-Surface-Pro-3:~/devel/vagrants/devstack-designate$ echo $DEVSTACK_JOB

We kick off the thing with vagrant up and voila, we have a Vagrant VM with DevStack running the job defined on the project/change we want to test :-) .

Happy hacking!

Be Sociable, Share!

Write a Comment

Comment