Is your instance not launching? Are your Floating IPs drifting aimlessly through the ether? Do you have a PI eligibility request? Chameleon tickets are the fastest way to reach the Chameleon support team and receive assistance for all your testbed needs. It’s 2020. Everyone could use a little extra help.
As 2021 and Oscars season approaches, the Chameleon team has compiled “Tickets of the Year” designed to help you avoid (at least some of) the same stumbling blocks of 2020. Read on to learn about some of the most common tickets, their solutions, and some special ticket award categories. You can always reach out to the Help Desk team for white-glove troubleshooting help.
Without further ado, the Oscar for the Most Often Encountered Error goes to… “No Host Found” Error!
Problem: Are you frequently encountering the “No Host Found” error when trying to reserve a lease? While COVID might prevent you from hosting a typical Christmas, with this fix, you’ll always be able to find a host on Chameleon.
Solution: This error occurs when all the nodes of the type you’re trying to reserve have already been reserved for the time frame that you want. While we can’t free them up for you, you can visit the Host Calendar, select the type of resource you want to experiment on, and see the time frames that are reserved and available. One important thing to note is the site that you’re reserving resources on - like UChicago versus TACC - as the sites have different resources and availability.
Category: Most Wanted: The Problem Report
Winner: The Help Desk Email: What Should I Include?
Problem: You probably haven’t been in an office since the beginning of 2020. Whether you’ve mastered email writing or still rethink every sentence, we’ve got you covered. Here’s everything you can include to help us resolve your problem faster and get you back to experimenting.
Solution: More information is always better! Attachments, error messages, outputs - include it all. That being said, here are some key points to make sure to include:
UUID of the Instance or Lease (if applicable): To find the UUID in the GUI, click on your Instance/Lease name and ‘Lease Detail’ will appear. The UUID is located below the ‘Name’, with the heading ‘Id’.
Just as a reminder, the Help Desk button is available on the Chameleon webpage, right next to your username. When you’re in the experimental portal (CHI@UC, CHI@TACC, CHI@NU), the Help Desk button is in the drop down menu after clicking on your username.
Category: The Most Avoidable Problem
Winner: Launching with Specialized Nodes
Problem: Chameleon offers a variety of hardware (explore the options); one common difficulty our users encounter are launch issues when using a specialized node of certain types: either GPU, FPGA or ARM64. Usually, users reserve the specialized node, but then try to launch a standard image rather than a specialized image. This normally leads to launch issues, and reports of being unable to ssh into the instance.
Solution: Luckily, there’s a quick fix for this difficulty. Simply use the specialized image that corresponds to the node you’re reserving.
For GPU: use CUDA specific images like CC-CentOS7-CUDA10
For FPGA: use FPGA specific images like CC-CentOS7-FPGA
For ARM64: use ARM64 specific images like CC-Ubuntu16.04-ARM64
Note: You may encounter a similar issue if launching a non-specialized node with a specialized image. For example, trying to launch a Skylakes or Haswell node with a FPGA image.
Category: The Largest Problem
Winner: Large Scale Network Deployment
Problem: In large-scale network projects, the experiment can be performance sensitive. However, the location of the nodes and racks that you’re using in your experiment can significantly impact performance, leading to asymmetric network behavior. If your experiment is performance sensitive, mitigating against this kind of behavior is crucial.
Solution: Use nodes that are in the same section of a rack. If you’re seeing asymmetric network behavior, it is likely because your nodes are at different locations on a rack. The bottom and the top of a rack each have their own switch, with an interconnect between them, which explains the difference in networking behavior.
To choose your nodes, use the host calendar under the Leases tab on your Chameleon portal page to identify free nodes in the same section of the rack.
As an example, if you wanted to create a lease with multiple nodes, you can adapt the following command (currently set for 2 nodes):
blazar lease-create --physical-reservation min=2,max=2,resource_properties='["or",["=","$name","<node_name_1>"],["=","$name","<node_name_2>"]]' <your_lease_name></your_lease_name></node_name_2></node_name_1>
Category: The Most Annoying
Winner: CloudFuse Breaking Up
Problem: Experiencing trouble with CloudFuse for object storage and migration? CloudFuse can sometimes have stability issues and will have trouble uploading large files (especially above 4GB).
Solution: One simple, quick solution is to try remounting:
Download your v2 or v3 RC file from the Chameleon web interface (click your username at the top right).
Create a file in your home directory (/home/cc/openrc) on the instance and copy the contents of the RC file into it.
Source the file:
$ source openrc
Now you need to mount the object store. It's usually better to unmount it first:
$ cc-cloudfuse unmount my_mounting_point
$ cc-cloudfuse mount my_mounting_point
While remounting is a good fix, if stability is important to you, try using Swift, the OpenStack Object Store. Though Swift requires more scripting knowledge, it’s more stable, and you can also automate with it! Check out their Large Object documentation to learn more.
Category: Easiest to Fix on Your Own
Winner: It’s Raining Errors: Leases in Error State
Problem: Are your leases falling into an error state? This can happen when an action isn’t applied to a lease, likely because it fails to meet Chameleon requirements, like an extension is too long. Luckily, there’s an easy fix to get it back up and running.
Solution: To reset your lease, it just needs to be ‘touched’. You can do this within the GUI or CLI with the blazar lease-update process:
In the GUI:
Click update lease
Set the name to what it already is
In the CLI:
Simply use this command:
$ blazar lease-update <lease uid=""> --name <current lease="" name=""></current></lease>
Category: Most Overlooked Fix
Winner: Connectivity: Pinging Your Instance
Problem: While 2020 is the poster child for connectivity issues, don’t let that be the case with your Chameleon leases! Look no farther if you’re having trouble pinging your instance or ssh-ing in.
Solution: Using Chameleon’s GUI interface, you can access a remote console to your instance. Once you’re there, you can check the console to see if instances have been deployed (and troubleshoot) or determine what failed. To access the console on your Chameleon portal page simply go to ‘Compute’ > ‘Instances’. Once you’re on the ‘Instances’ page, click the instance name and select the ‘Console’ tab. It may take a minute to load, but then you’ll be able to troubleshoot your connectivity issues yourself, and see why you can’t connect to your instances!
Category: Commonly Misused Feature
Winner: PI Requests
Problem: While it's tempting to want to be a PI - and have all the power and responsibility that accompany it - there are a few logistical requirements to check off first.
Solution: To be a PI, you must be a member of one of the following groups:
A faculty member of an academic institution
Research staff of a federally funded lab, center or agency
Conducting research at a museum, observatory, library etc.
NSF Graduate Student Fellows
State educational offices or organizations
Each group is explained more fully in the Chameleon FAQ. If you believe that you are an exception, Chameleon occasionally provides exceptions on a case-by-case basis.
One last reminder before Chameleon signs off for the holidays, the Help Desk button is available on the Chameleon webpage, right next to your username. When you’re in the experimental portal (CHI@UC, CHI@TACC, CHI@NU), the Help Desk button is in the drop down menu after clicking on your username. No ticket is too small, no problem too complicated, and now you have everything you need to write the perfect Help Desk ticket.
That’s all, folks. We’ll see you next year!