Compliance Goes Virtual

VTM Group’s Engineering Technical Services (ETS) provides services to 15 of our association clients. With 11 full time engineers on staff, and specialties in compliance, program management and certification, our clients engage ETS for management of compliance and certification programs, authoring specifications, developing test procedures, overseeing independent test labs, and more.

Many organizations and individuals in the high-tech association space are familiar with the PCI-SIG and the USB-IF. Both organizations have mature and robust compliance programs that have been in place for 25+ years. VTM’s ETS team serves as the ‘technical experts’ for both of these organizations, providing the physical testing of vendor products, adherence to technical specifications, and overall compliance program management.

One of the most visible roles the ETS team provides for both of these organizations, as well as several others, is the testing of vendor products at in-person compliance workshops. A compliance workshop is usually held at an external venue or hotel during a 4-5 day period. Vendors and their products attend these events to have their products tested in gold suites which contain compliance tools that analyze, measure, and report if products adhere to the organization’s technical specifications. Attending a workshop provides the vendor with important information about their product’s compatibility and readiness for the marketplace, as well as the potential to be listed on the Integrators List, or obtain a certification logo.

As with many organizations, the recent impacts of COVID-19 have made it difficult to hold in person events. Typically, PCI-SIG and USB-IF each hold 4-6 in-person compliance workshops per year, with events in the US and Asia accounting for up to half of the 15-20 Plugfests/Workshops that VTM supports in a typical year. Given the importance of compliance workshops to members and product development, it was imperative that the ability to test and certify products continued. ETS had a solution for that.

Our Vision

Within a matter of weeks, members of the ETS team developed and proposed the concept of a virtual workshop. After many brainstorming sessions on what we could make possible and how to provide the best experience for members, client specific plans were presented to PCI-SIG and USB-IF. In less than 3 months the VTM team brought the idea of a virtual workshop to life and redesigned 25+ years of in-person workshops.

The overall virtual workshop concept leveraged the VTM building and staff. We were able to utilize the 2^nd floor of our 28,600 square foot building to designate conference rooms for each test set up. VTM staff from ETS, as well as our Association Management and Event Management teams, managed the product schedule and transition between testing rooms. It was important that members were able to participate remotely, if desired, which VTM was able to accommodate via real-time WebEx or Zoom sessions. Members were also provided with immediate access to their test results, so they could monitor and address any testing issues.

Planning and Execution

With the initial concept defined, we got down to making it happen. We considered EVERYTHING.

• Electrical: VTM’s Facilities team mapped out the building’s electrical circuits to ensure we could accommodate large power needs.
• Internet: VTM’s Corporate IT team established a new wired and wireless network that was dedicated to the event. The new network was separate from the main bandwidth and allocated specifically for the workshop. Special network drops were added to all rooms used for testing. Global IP addresses were created outside VTM firewalls for vendors who needed special remote access to their product during testing.
• Test Schedule: ETS determined how to maximize efficiency of testing for both staff and members. PCI-SIG opted for a 2 week test schedule, which allowed different types of testing and equipment set up each week, whereas USB-IF opted to test 1 specific product category during a single week. Interoperability testing was conducted in a large testing room with 8-12 systems simultaneously managed.
• Test Equipment: In order to replicate the Gold Suite testing from an in-person workshop, it was necessary for the majority of PHY test equipment to be brought in. PCI-SIG coordinated support from PCI-SIG approved domestic gold test equipment vendors. These vendors attended during the 2nd week of testing. USB-IF leveraged local support from the USB PIL to provide needed equipment and additional testing expertise. With these contributions from the test equipment vendors, members were able to receive the same high-quality testing as an in-person workshop.

USB-IF testers in each of their private testing areas.

• Coordination with product owners: Product owners were asked to ship their product to VTM prior to testing. Once on-site, products were pre-tested for functionality, inventoried, and secured. ETS worked directly with product owners to confirm custom configuration needs, remote control needs, and special support requests. VTM staff managed the transition of products between test suites and communicated with product owners as needed on product configuration, test results, and troubleshooting.
• Member Participation: To ensure members were able to participate virtually, VTM created virtual audio and video web conference hosts for each test suite and provided the submitting vendor with a detailed test schedule in advance with attendee links for every scheduled compliance test. Over the three weeks of testing, over 300 individual video conference sessions were scheduled between compliance testing rooms and the scheduled device vendors. In addition, real-time test results were synchronized to the organizations’ compliance servers for vendors to track their individual pass/fail progress for each compliance test and stay in touch with progress and ask questions.  

One of the many PHY test sessions conducted for the workshops, the virtual session is visible on the laptop to the right of the test engineer.

USB-IF virtual test screen share from the vendor perspective.

• COVID safety and protocols: We took COVID measures very seriously. Attendees in the building were limited to essential staff only. Upon arrival at the VTM building, attendees utilized an automated process to answer COVID screening questions, followed by a temperature check. After a successful screening, attendees were admitted to the building. All staff wore masks, equipment and common spaces were sanitized and wiped down regularly, building signage regarding social distancing was plentiful, and professional cleaning of the building was increased.

 A few of the many COVID precautions - automated check-in and screening process, reminder signage about social distancing, mask usage, and prevalent hand sanitizer.

• Staffing: Given COVID measures, we were thoughtful about staffing numbers and participants. We were able to leverage VTM staff for all elements of the event, with the exception of the electrical test equipment vendors who participated during the second week of the PCI-SIG event, and individuals from USB-IF’s Platform Integration Lab (PIL). Any non VTM employees adhered to VTM’s COVID safety measures the same as employees.
  
• Meals: To keep our staff energized, coffee, lunch, and snacks were provided. All lunches were individually wrapped and labeled. Water bottles and individually wrapped snacks were also available.

“Our compliance workshops are an invaluable USB-IF member benefit and critical to enabling a robust, reliable USB ecosystem that consumers rely on. When the global pandemic presented a series of challenges for our traditional workshop, VTM and the USB PIL team stepped up to the plate, provided a comprehensive virtual alternative, and executed that plan seamlessly. While we look forward to holding in-person workshops in the near future, we are proud to continue offering this member benefit in support of our mission to deliver on the USB-IF brand promise.” ~ Abdul R. Ismail, USB-IF CTO and Compliance Chair

Outcome

While the virtual workshop lacked the ability for members to network with each other, it accomplished member’s testing needs and desires. Add-in card products received the same exposure to interoperability systems as an in-person event, but testing was done more efficiently in 15 minute increments rather than one hour time slots.

Virtual vendor participation throughout the event was high. Members were appreciative of the opportunity to engage directly with the testers, watch the tests and be able to troubleshoot any issues, gain valuable information about their product, review test results immediately online, as well as save time and money by not having to travel or be out of the office for a full week. We received a steady stream of emails throughout the event from vendors who were monitoring their products and sending in questions.

“PCI-SIG compliance workshops are essential for preserving and growing the PCIe technology ecosystem. VTM Group serves as a trusted partner and due to their impressive innovation, we are able to continue offering this key membership value even in the midst of these challenging times.” ~ Al Yanes, PCI-SIG President and Board Chair

Virtual screen example from the PCI-SIG CV Gold test suite, as seen from a vendor’s perspective.

What Did We Learn?

The virtual conference sessions were informative for both sides. Since the time commitment for product owners was less than an in-person workshop, our team had the opportunity to engage with a person familiar with the product, who may have otherwise been unable to attend an in-person event. The ETS team was able to accommodate specific product handling and technical management of products on behalf of the companies.

Although we didn’t anticipate the large amount of time needed to configure products for interoperability testing, or their unique requirements (some came with an entire manual!), the ETS team once again rose to the challenge. They invested long hours becoming familiar with the product, as well as participating in phone calls with product owners to help troubleshoot product issues. We saw this as a value-add to both our clients and their members.

Multiple test rooms were used to ensure confidentiality of testing and acccomodate social distancing of Test Equipment vendor teams.

We identified small logistical items that could help the workshop run more efficiently next time. These efficiencies may help us accommodate more products in the future – which is great news, since we had a lengthy list of products we weren’t able to accommodate this time around.

We have the ability to scale virtual test events as needed. We anticipate higher demand for virtual testing opportunities as COVID continues and have the space, staff, and experience to accommodate different testing models and durations. 

In closing, and on a personal note, we’d like to thank all the VTM team members who made these events possible for our clients. We appreciate their problem solving, creativity, dedication, and hard work!

USB-IF
Pictured: Rami Alshafi, Travis Wineinger, Chris Brokke, Jaime Grimm, Angela Hardy, Jessica DeGennaro
Not Pictured: Mark Paxson, Zach Uhles

PCI-SIG
Pictured: Joe Leong, Travis Wineinger, Adrienne Foster, Kaitlyn Gooding, Robby Crips, Chris Brokke (front)
Not Pictured: Doug Gemmill

 If your organization is considering a virtual test event and would like to learn more about VTM’s ability to support these events, contact us at eventinfo@vtmgroup.com.