CHAPTER 4

Case Studies

Introduction

This chapter presents five illustrative case studies documenting the experiences of state departments of transportation (DOTs)—from Nebraska, California, Iowa, Missouri, and Delaware—in developing and implementing machine learning (ML) solutions to address different needs, including asset management, safety, and transportation systems management and operations (TSMO). These examples can help state DOTs learn from existing applications that are ready for near-term deployment, better evaluate and understand the capabilities offered by ML, identify the resources and skills needed to implement ML methods and provide insights into costs, benefits, and limitations of ML-based solutions. The information for these case studies comes from interviews conducted with agency representatives (e.g., project managers and data scientists/engineers) or consultants who have developed or deployed ML solutions. The interviews aimed to capture information on challenges that agencies faced in their deployment process, their experience and satisfaction with ML applications, and any lessons learned that could benefit other deployers.

For the interviews conducted with the state DOTs, a structured set of topics and questions guided the discussions, outlined as follows:

1. ML Application: How has your department applied ML in transportation?
   • What transportation challenges are being addressed with ML?
   • How did your department decide that ML would be a good solution for this transportation problem?
   • At what stage of deployment is the ML solution?
   • Did your department use in-house, open-source, and/or proprietary tools/products?
   • Where and how large is the deployment (e.g., # of miles, # of intersections, # of users)?
2. Data: What data support the ML task?
   • What are the data challenges with the ML: quality, quantity, privacy, latency, storage, labeling, and ownership?
   • Is proprietary data needed?
   • What is the cost of data collection and storage?
3. Costs: What is a rough cost estimate for the ML application?
   • How might these costs change over time?
   • How long did it take to deploy vs. what was the expected deployment schedule?
4. Evaluation: What performance metrics are being used?
   • Is there an evaluation plan? If so, what is included in it?
   • What is the main benefit of this ML application?
5. Workforce Capacity: What skills, capabilities, resources, and departmental capacities were necessary for the implementation of this ML solution?
   • Were consultants contracted for ML expertise?
   • Did your department hire internally for ML expertise?
   • Was training provided to development/deployment personnel?
   • Was training provided to operational personnel?

6. Collaborators: Is your department collaborating with others (e.g., purchasing vendor ML products, working with a university or other agency) for the ML application?
7. Stakeholders: How were key stakeholders identified? How was stakeholder buy-in secured? How does your department continue to involve stakeholders?
8. Lessons Learned: What challenges or issues did your department face during implementation? What are some of the most important lessons learned? Any roadblocks for moving forward with implementation?
9. Next Steps: What next steps are planned with this project? (e.g., scaling up)

The level of detail the interviewees provided in response to these questions varied for a few reasons. First, their levels of familiarity with different aspects of the projects were different. In addition, some data elements (e.g., the cost of the ML application) might not be easily obtainable for complex projects involving multiple departments or phases. Furthermore, some information (e.g., the specific type of ML method used) might be proprietary as the DOTs commonly outsource technology projects to consultants and vendors. Finally, interviewees differed in their preference to share certain information. Therefore, the summaries presented in this chapter for each case study contain a varying degree of detail for the set of questions presented above.

The case study descriptions presented here are mainly based on the information gathered from the interviews. In some cases, the interviewees provided documents or weblinks for additional relevant information for the ML applications they deployed. The interview transcripts as well as the video recordings were used in summarizing the information for the case studies. To the extent possible and as applicable, the information from each case study is summarized under common subsections that include: (i) Introduction/background (ii) Machine learning applications; (iii) Data storage and collection; (iv) Organization, workforce, and stakeholders; (v) Evaluation; and (vi) Lessons learned.

The interviews were conducted in December 2022 and early January 2023. Table 20 shows the list of five DOTs interviewed and the types of applications for which they used ML methods and technologies. All five agencies implemented solutions involving deep learning (DL), primarily using image or video data. In addition to DL, Missouri DOT implemented unsupervised learning as well as a boosting method. Boosting is an ML technique that enhances prediction accuracy by combining multiple weak learners (e.g., one-level decision tree) into a single strong learner, focusing on incrementally correcting misclassified examples from earlier models. The following sections delve into each case study, providing additional details.

Nebraska Department of Transportation (NDOT)

The Nebraska Department of Transportation (NDOT) had two proof-of-concept projects to extract roadway asset information from image data. In the first project, the goal was to locate and classify guardrails and guardrail attenuators on NDOT’s network using ML. The second one was on detecting marked pedestrian crossings from geo-encoded images collected by NDOT’s pavement profiler van. In both projects, Convolutional Neural Networks (CNNs) were used for object detection. For the first project, classification algorithms were used to determine attenuator types and sub-types (i.e., endcaps). This case study details the experience of NDOT in managing this deployment, including their approaches to data and infrastructure, workforce and stakeholders, and evaluation, as well as challenges and lessons learned.

Guardrail Detection

In this pilot project, NDOT explored the viability of using ML for creating a system-wide inventory of roadway assets on the state’s highway and interstate systems. More specifically, existing geo-coded images collected by NDOT’s pavement profiler van were processed and relevant ML/DL methods were built to detect guardrails and classify them by type. NDOT hired a consulting firm to complete this pilot project as they did not have the in-house expertise needed for building ML models. The firm processed the images in Google Cloud and used virtual machines with NVIDIA GPUs for model training. They labeled about 1,500 images that contained guardrails and guardrail attenuators for model training purposes. A Convolutional Neural Network (CNN) model coded in Python was used and trained for object detection, i.e., detecting guardrails. The CNN model was applied to roughly 2.5 million images from NDOT’s 2019 roadway network profiling to create a statewide inventory. Figure 7 shows a sample image and the result of the object detection model.

The bounding boxes containing the guardrail (as in Figure 7) were cropped to create images that were passed to an object classification model which was trained to distinguish among three types of guardrails (i.e., crash cushion, Type 1, or Type 2 attenuator). If the image is classified as a Type 1, it is then sent to another classification model that is trained to assign a subtype to distinguish among five endcaps. The results from the object detection and classification models were used to create a GIS layer to locate relevant guardrail information as points and line features on a map.

Marked Pedestrian Crossing Detection

The goal of this pilot project was to identify the locations of all statewide marked pedestrian crossings on the urban highway system from roadway imagery data – sample in Figure 8. A hybrid approach was employed where the painted crosswalks are first detected by a DL object detection algorithm (YOLOv5) from an image. The results are then inspected and reviewed manually for crosswalk marking type assignment (e.g., standard, zebra). Overall, a total of 1,303 marked crosswalks were identified on the state’s urban roadway network. To train and evaluate the DL model, sample images were manually labeled by drawing bounding boxes around the marked crosswalks visible in the image. The predictions from the DL

model (also bounding boxes) were then compared to manual labels to assess model accuracy in terms of detection rate.

Data Storage and Collection

As indicated above, for both projects NDOT relied on video logs collected by the cameras on the profiler van. There were several challenges in processing large volumes of video data. One challenge was the large size of the video log files, which made it difficult to transfer files. These large files were downloaded to a hard drive and shipped to the vendor for processing. The vendor did any required image labeling for model training and testing. The scope of the project did not include categorizing the crosswalk types automatically by the ML/DL models. The ML models were primarily used to identify only the marked crossing’s location, i.e., detecting its presence in an image. The project team felt that this was sufficient for the low sample size of about 1,300 locations across the state. Categorization was more complex and done manually as the sample size for different types of crosswalks was not large enough to train robust ML models.

Organization, Workforce, and Stakeholders

Like many other state DOTs, NDOT lacked the needed in-house expertise in ML/DL to undertake the projects discussed above. However, by working with consultants, NDOT was able to successfully implement the two ML applications. They are also collaborating with a university on ML-based solutions for other types of applications. For example, rather than manually drawing crash diagrams, they are working with students from a local university on creating such diagrams for intersection crashes using Google Earth images. For this application, they spent about $50K on the student project to create the necessary ML process. The method was not fully implemented yet, but it was partially functional. In addition to these collaborations, NDOT has business intelligence staff that are getting up to speed on ML. They have the capacity to replicate the consultant’s work by employing the ML source code provided by consultants for the completed pilot projects. Building in-house capacity is critical to accomplish more with ML methods, rather than having to rely on contracts to go through to hire consultants for each project.

The Nebraska DOT’s executive leadership was initially skeptical of ML but became supportive after hearing the potential benefits of automation in data collection, cost savings, and improved accuracy of ML methods over time. The leadership was hesitant to invest in the new technology, citing the risk and cost. However, after seeing the successful completion of the guardrail endcap project and the marked pedestrian crossing inventory project, the leadership became more trusting and open to investing in machine learning. Success stories were helpful in garnering support for investing in workforce development in ML and a better understanding of this new technology.

Evaluation

To evaluate the performance of the ML applications, the consultants relied on sample labeled data with known answers. To reach an acceptable level of accuracy, they performed multiple rounds of model training. For the guardrail detection application, the ML models achieved accuracies of 97% and 85% in detecting the presence of guardrails and classifying them into the three types (i.e., crash cushion, Type 1, or Type 2 attenuator), respectively, when tested against the validation dataset. At the same time, the downstream Type I classification model for distinguishing among five endcaps achieved an accuracy of approximately 95%. For the marked crosswalk detection project, on the other hand, the consultant tried to achieve a reasonable balance of false positives and false negatives and get good true positives. They also manually reviewed and validated the location, accuracy, and completeness of all crosswalk points and assigned a type to each using ariel imagery to pinpoint crosswalk locations. There was no official evaluation plan in place to assess the level of accuracy of the ML models.

Both projects were quite cost-effective for NDOT, each had a budget of $25K, as they were able to capitalize on existing image data. In addition, each project had a well-defined scope with clear goals that helped simplify the execution of the project.

Lessons Learned

From the onset of the two pilot ML projects, the NDOT team had a focused approach to clearly define project scope and goals. These two initial implementation projects (guardrail and marked crosswalk detection projects) were successful due to their limited scope with well-defined goals. Also, NDOT staff had a good understanding of the limitations of the technology and set a relatively low threshold for success. To get the buy-in from the NDOT leadership and administration, demonstrating success was critical for investment in any new technology. Therefore, the project team focused specifically on low-cost and low-risk applications with clear foundational benefits. They intentionally avoided more complex ML applications with large project budgets, e.g., inventorying every asset and spending millions of dollars. The team suggests caution in adding complexity to the project as it increases the risk of not achieving the desired success. Starting with a simple, concrete project with a known answer and low investment is recommended before investing large sums of money.

The team learned that while having a free image dataset from an existing source was beneficial, there were also risks involved, such as degraded asset conditions (e.g., faded crosswalk paint) and incomplete data. It was also important to identify risks from both a technology and success perspective. For example, as the profiler van is driving through an intersection there may be crosswalks on all four legs, but the cameras may capture the images of only two of them, the two that it is driving over, not the other two on the crossing road. Therefore, the team decided to focus only on those crosswalks directly visible from the profiler van’s path and leave the others out for the time being. Such decisions were made early on to mitigate potential risks from a success perspective.

Overall, the project team demonstrated the importance of careful planning and a measured approach when implementing ML technology. By keeping the scope limited, identifying potential risks early on, and

starting with a simple and concrete task, the team was able to achieve success with the two ML projects and demonstrate the value of the ML to leadership and stakeholders.

NDOT has plans to build additional capabilities using ML methods to support the agency operations. For example, identifying fixed objects along the roadway and estimating their proximity to the travel lanes are important inputs for safety assessment and crash risk modeling. They are exploring applicable ML methods to document such objects from image data so that the density of potentially hazardous roadside objects per mile can be quantified for the highway network.

California Department of Transportation (Caltrans)

The California Department of Transportation (Caltrans) has diverse needs across multiple areas, including asset management and traffic data collection. Several initiatives and projects are underway within the department, such as the litter abatement program, right-of-way assessment, traffic counts, and asset condition monitoring and maintenance. To address these needs, Caltrans has leveraged different AI/ML applications, specifically computer vision. One example is the PathWeb system, which uses computer vision-based data collection and provides data access through a web portal. Additionally, the maintenance and traffic operations divisions have collaborated with Google on separate occasions to use their street view imagery for turnkey operations, like those conducted with PathWeb. While these partnerships with external companies and consultants have been successful, Caltrans recognizes the importance of developing in-house capabilities to perform such tasks and avoid potential issues that can arise with commercial solutions, which may lack transparency. To that end, the planning and mobility division is establishing a center of excellence for data analytics, including practical ML approaches, to enable staff to move beyond mainstream applications like computer vision and develop in-house expertise in the analytics space. One of the primary challenges is articulating the significance and potential of AI/ML to departmental managers and executives, essential for gaining their support and understanding.

One of the main motivations for using AI/ML was to streamline the data collection process for environmental assessment and asset management which can be very time-consuming and costly. Manual data collection is also prone to human error. There are several examples where Caltrans worked with vendors for these types of tasks:

• Caltrans works with PathWeb to monitor roadway assets network such as median barriers, guardrails, signage, boxes, etc. They collect asset data, and in the backend, ML is used for object detection as well as condition assessment. PathWeb has a similar solution for pavement condition assessment but uses a different type of camera. PathWeb has a statewide contract with Caltrans for a period of 5 years.
• For a pilot project, traffic operations explored the use of Google Street imagery but were concerned with some privacy issues.
• Caltrans worked with a vendor on a project related to intersection safety monitoring using data from traffic cameras. They wanted to detect near misses and the quantity of potential crash risks.
• As part of a pilot study, Caltrans is also working with Payver, another company that has a solution for streamlining litter assessment and data collection. They are using dashboard imagery instead of a van with a camera and GPS to collect imagery on selected roadways on a limited basis. In summary, they record the frames and analyze them before they turn the raw image data over to Caltrans. One of the main advantages of this company is that their application is built on Caltrans’ adopted Esri platform.

Data Collection, Storage, and Quality

Caltrans’ new data collection initiatives are designed to enhance existing efforts, particularly in the face of the significant labor costs associated with gathering the vast amounts of data required for video imaging applications, whether via cars, vans, or drones. Data storage and maintenance costs could also be significant. Caltrans considers two primary strategies for image data collection:

The first approach is to collect image data specifically when and how it is needed, which can be flexible and highly accurate but also costly. This approach can be tailored to the exact needs of the specific task at hand, but the cost of data collection can be substantial.

The second approach is to purchase existing imagery from contractors or vendors, which can be relatively inexpensive compared to the first approach, but it sacrifices the flexibility and accuracy of collecting custom data. Caltrans has explored purchasing a package for image data collection and processing from Google for the litter detection program. However, based on feedback about the cost of such a contract, it was apparent that its cost could easily grow to several hundred thousand dollars over time. In addition to the cost of data-related tasks, storage costs for many gigabytes or terabytes of image data can also be substantial, even if Google Cloud is used instead of dedicated in-house data servers.

On‐land Visual Trash Assessment (OVTA) pilot is a good example of the use of AI/ML with image data. The goal of the OVTA pilot is to use OVTA photos for trash detection (object recognition) and trash level rating (classification as low, moderate, or high). Object detection uses a trained deep learning algorithm. Object classification is based on decision trees and a random forests approach which used if/then statements to arrive at a trash level rating for a single photo.

Evaluation

Currently, the approach to evaluating the performance of AI/ML methods is conducted on an ad hoc basis, indicating that these assessments are carried out without a standardized procedure. It involves validating the outcomes of these pilots which employ AI/ML technologies by the DOT staff. Currently, there is no sophisticated benchmarking and metrics used for evaluation purposes. For vision-based solutions, it is conceptually easier to evaluate, like in the case of litter detection, because the presence of trash in the image can be ascertained by visual inspection. For the pavement condition assessment application provided by PathWeb, the metrics are compared to those from manual methods. Although this manual approach can be time-consuming and relatively subjective, it is in fact easier than using various analytical results presented in a table because people understand what they see better than what the summary statistics (e.g., accuracy, recall, precision, false alarms) tell them. The ability to directly evaluate the accuracy of vision-based methods is instrumental in their adoption as primary AI/ML methods for various applications. This is because accurate evaluation allows for the identification of potential areas of improvement and can help build confidence in the reliability and effectiveness of these methods.

An example of a more methodological approach for the evaluation of AI/ML technologies is given for the On‐land Visual Trash Assessment (OVTA) pilot. Both detection and classification approaches were evaluated. The detection model was found to be correct 50% of the time in terms of detecting actual trash. Overall performance of the detection model was also around 50%. For the classification model, 70% of the time model rating matched the human assessor’s rating.

Overall, benchmarking for evaluation purposes remains to be a challenge. Even a simple result such as 80% accuracy in the case of computer vision is not very easy to explain. If slightly more technical evaluation metrics such as “precision” and “recall” are used then it becomes a real challenge to explain to people who are not in the field of AI/ML, such as regulators or managers.

Workforce capacity development

The main area of focus in workforce capacity development is data analytics with an emphasis on extended data usage and data literacy. There are traditional GIS tool developers that incorporate data analytics and ML functionalities into the basic user interface so that the user can perform data-driven analysis in GIS. Incorporating these into existing tools and being able to train DOT users will expedite the learning curve for AI/ML methods and help build workforce capacity.

Caltrans currently has limited in-house expertise in data science and machine learning, with only a small number of staff capable of performing programming tasks. At this point, there is no official staff classification as “data scientist” and/or “AI/ML engineer.” Within the planning division, the integrated transportation program pursues some relevant efforts including a plan to consolidate all public transit operations using the public data feeds and to upgrade the pavement management system. Within this program, they hired a few staff members with programming capabilities to work on these initiatives. Caltrans plans to pursue this model and hopes to hire more staff with data science expertise in the future.

On the industry (vendors and consultants) side, the amount of collaboration is limited due to not having in-house staff and understanding of the necessity to get them engaged. However, a considerable amount of work is being performed in terms of setting up some pilot projects mentioned above as well as establishing an open data portal with an enterprise data catalog that facilitates data sharing. Overall, staffing is a bottleneck, and the organization needs staff with the necessary skills to deploy, edit, and maintain the AI/ML applications.

Lessons Learned

There are several lessons learned that can be summarized as follows:

• If ML is used, then there is a need for regulatory compliance to obtain approval for its adaptation as well as for explaining/interpreting the results. In this context, a major lesson learned is the importance of being able to communicate the meaning of the accuracy of these methods to people outside the AI/ML field including regulators, managers, and executives. If this is not streamlined and not done right, it will be difficult to adopt AI/ML technologies.
• The experience with the environmental litter project showed that it is better to work with vendors that specialize in ML/AI technologies rather than traditional civil and environmental engineering consultants. This is because ML/AI technologies are relatively new and rapidly evolving, so specialized vendors are more likely to have the expertise and experience necessary to develop effective solutions. Another advantage of working with companies specialized in AI/ML technologies is that they are very flexible and willing to and capable of trying different approaches or modifying their existing solution to address different problems suggested by the DOT. This makes it easier to solve various problems as the DOT encounters them, a big advantage that might not have been feasible with traditional architecture/engineering companies based on past experience.
• As AI/ML technology continues to mature, we can expect specialized companies to offer more custom solutions that can help streamline turnkey projects that can be integrated into existing systems and workflows. By working with these companies and vendors, DOT can benefit from their expertise and access to these custom solutions, which can save time and resources while also improving the accuracy and effectiveness of the AI/ML models. It’s important to stay up to date on these developments and work with vendors who are committed to staying on the cutting edge of AI/ML technology.
• It is crucial to demonstrate the benefits of AI/ML technologies to executives, as their support is essential for adopting and integrating these innovations. One effective way to illustrate this point is by using computer vision as an example. Since computer vision is based on AI and operates through visual processes, it is easier to explain and demonstrate its advantages to both executives

and technical staff. Therefore, having clear and concise communication strategies that effectively showcase the value and benefits of AI/ML across all organizational levels will be key to their successful adoption and integration.

• It is important to understand the long-term costs and benefits of procuring AI/ML technology from external sources versus developing solutions in-house. Internal development can provide more control over the process, as well as lower recurring costs. However, it is also important to consider the expertise and resources required to develop and maintain these solutions in-house. It may be more cost-effective to rely on external vendors for certain tasks, especially if they have specialized expertise and experience in implementing AI/ML solutions. In any case, DOT staff should have a solid understanding of the technology and its application to ensure that the best long-term business decisions are made. This includes evaluating the appropriateness of specific AI/ML technologies for the problem at hand, as well as assessing data quality and other factors that can impact the success of the project.
• A tight integration with the IT department is also needed to plan and maintain the most efficient communication network and computational infrastructure required to deploy these AI/ML technologies. As AI/ML solutions typically require significant computational resources and efficient communication networks to function properly, effective coordination between the transportation departments and the IT department is essential for the successful deployment of AI/ML technologies.

Next Steps

Over the past five years, Caltrans has conducted several ML pilot projects primarily aimed at improving operations. Concurrently, they established a data governance plan in 2017, which led to the development of a data catalog and the enhancement of data storage capabilities. Looking ahead, it is anticipated that AI/ML technologies will be increasingly utilized in transportation planning and may become integral to business processes within the next five years. Key to this broader adoption will be the ability to recruit new staff with expertise in AI/ML and data science, as well as the demonstration of success in current pilot projects. Moving forward, Caltrans plans to integrate AI/ML technologies with existing DOT tools, similar to the deployment of dashboards in Microsoft Power BI over the recent years.

Iowa Department of Transportation (Iowa DOT)

The research project team interviewed a faculty member and a research engineer at Iowa State University who have been involved in transportation research funded by the Iowa Department of Transportation and have developed ML applications in traffic operation and safety, and data management and visualization. The research projects included highway performance monitoring and incident detection applications. The applications were developed to improve highway safety and provide better management of traffic incidents. Data collected from video, traffic volume, speed, backups, weather, and more is being stored from across the state in the Real-time Analytics of Transportation Data Lab (REACTOR) and analyzed to enhance the ability to respond to traffic congestions in real-time. The incident management system aims to minimize the impact of single incidents. The applications also turned to artificial intelligence methods to help with data mining, as well as understanding driving patterns. The research team continues to tap into big data analytics to help Iowa DOT improve the safety and efficiency of the transportation network.

One of the applications involves using ML for calculations of some performance measures and for alerting DOT of non-recurrent conditions. The university researchers use video data and other data sources to provide statewide performance assessments for the interstates, including the identification of hard brakes and their causes. ML models were initially considered for this application, but such models were considered

too costly for statewide deployment. This prompted the team to revert to simpler local detection methods instead. The team also considered ML methods for the detection of anomalies but resorted to simple threshold-based methods to identify erroneous detector observations caused by detector malfunctions. The decision was made to avoid the high cost of running ML models in the cloud and consequently, the high cost-to-benefit ratio.

In summary, the research team attempted the use of ML models for incident detection using camera analytics, incident detection in work zones, detection of recurrent and non-recurrent traffic conditions, and detection of anomalies in traffic data. Due to the high cost of upscaling such applications for statewide deployment, the researchers decided to adopt simpler models instead.

The researchers are also aware of an ongoing testing project for snowplows where data is fused from multiple sources to detect an approaching vehicle before hitting a maintenance vehicle. The project involves testing several trucks that are retrofitted with sensors and using ML for object detection before a maintenance vehicle is hit by another vehicle.

DOT is currently evaluating technologies for incident detection using video cameras. The university researchers are currently working on an evaluation prototype project for wrong-way detection and anomaly detection using cameras. Another ongoing project involves the detection of the onset of incidents using shallow models, as opposed to deep learning models.

Data Challenges

The researchers identified a few challenges that are associated with the deployment of ML applications in the real world. While development and testing of ML applications with limited data sources (e.g., video cameras and sensors), the major challenge remains when such applications are deployed for large-scale implementation at the statewide level. Such large-scale applications may also require high-performance computing, in addition to other issues related to maintenance and the need for more bandwidth to support real-time access to large amounts of data from sensors and cameras. This poses a challenge for such applications to have continuous access to all the data needed to run successfully. The researchers also pointed out that data quality from such a vast number of cameras and sensors is not guaranteed, and therefore, remains a major concern for achieving good prediction and high performance of such applications. In some applications such as incident detection, a prediction accuracy of 99% may not even be acceptable by DOT and traffic management agencies due to the large number of false alarms produced when applications are deployed at a large-scale level. This triggered the need to apply post-processing methods to reduce the number of false alarms sent to DOT.

Development and Deployment Cost

Cost is significant at the production system level in terms of deployment. However, the development cost is not considered significant from the researchers’ standpoint. Cost also involves making sure the code is at least 99% reliable and this is impacted by where the code will be running and the possibility of power failures that would impact the entire system operation. Therefore, it would be best to run the code in the cloud which adds substantially to the overall deployment cost. There is also a cost associated with the management of the code after the system is up and running in the real world.

Benefits and Performance Metrics

For the work zone project, the team used the number of false alarms as one of the performance metrics, in addition to recurrent meetings with the management team to seek feedback and address other issues related to performance. But overall, there is no formal system used for evaluation yet because the full ML

operation has not been implemented yet. Also, the research team closely tracks the operational cost of the incident detection system.

Involvement of Stakeholders and Collaborators

The research team did not involve outside consultants in the development stage. When the system went into the deployment stage, the team handed off the operation to Iowa DOT and continued to provide some support. The Iowa DOT later contracted with a consultant who now pulled the API and is handling the notifications to the respective districts and other stakeholders who are interested in these messages. There are multiple agencies and vendors when it comes to receiving data. The vendors provide Advanced Traffic Management System (ATMS) support for the agency. At some point in time, DOT switched vendors, and this affected the feeds used to inject the data and triggered the need to go back and rework the ingestion process. Some vendors followed the standards for data distribution, but there were slight differences that were discovered later. This impacted the ML process because of the data feed itself. As far as the training support is concerned, there was no formal training on the end user side beyond providing basic demonstrations of how to receive alerts and how the system works. There were no other collaborators involved.

Lessons Learned

There are a few lessons learned. One important lesson is to have early wins for DOT to see value in the models developed regardless of the approach used. It is very important to provide assistance in the decision-making process as one of the top priorities for DOT, even if the models are simple threshold-based. This might be all they need to support their decisions.

The second lesson is to maintain continuity when multiple consultants and vendors are involved. This helps ensure compatibility over the long run and increases familiarity of all parties involved with the system as it evolves, reducing the amount of effort needed to address operational and management issues along the way.

The third lesson is that data is hardly as clean as we expect it to be, and this poses a challenge. Vendors may not always provide all the information needed to address this issue. For instance, the clarity of images from video cameras tends to be affected by the wind and the team had to find a workaround for that. Checking the data quality is a time-consuming process, but it is necessary for the model performance. There are also other challenges that come up over the course of implementing the model or system. Challenges appear when it comes to scaling up the system, not to mention the transferability and robustness which are not often tested with ML models. This is why sometimes simpler models tend to be more transferable and more convenient to use.

Overall, there is a need to create a good pipeline and a robust evaluation system that allows new models to be tested and compared against old models in a timely manner to expedite the evaluation and implementation process.

Missouri Department of Transportation (MoDOT)

In 2020, The U.S. Department of Transportation’s Federal Highway Administration (FHWA) awarded a grant of $1 million to the Missouri Department of Transportation (MoDOT) as part of the Advanced Transportation and Congestion Management Technologies Deployment (ATCMTD) program for the Predictive Analytics for Traffic Management on I-270 in St. Louis project. This project deployed a predictive analytics platform that integrates information on traffic volumes, weather, and special events to determine the likelihood of crashes and improve response times (FHWA 2020c). This case study details the

experience of MoDOT in managing this deployment, including their approaches to algorithm implementation, data, infrastructure, workforce and stakeholders, and evaluation, as well as challenges and lessons learned.

Background

MoDOT was interested in studying factors that cause crashes along their road network, with the overarching goal of reducing crashes. They knew that a wide range of factors can lead to crashes, from driver behavior to roadway geometry to weather conditions, leading to seemingly random occurrences of crashes.

MoDOT applied to the ATCMTD grant program to deploy three technologies to help facilitate a proactive approach to safety, congestion, travel time, and emergency response time. This pilot deployment is located primarily on Interstate 270 (I-270) in the St. Louis District. This makes MoDOT among the first state DOTs in the country to focus on predictive analytics in a heavy construction area (Missouri Department of Transportation, 2022).

I-270 is a very urbanized zone with many crashes that occur at seemingly unpredictable times and locations. Additionally, MoDOT knew that there were going to be ongoing work zones in the area, elevating worries about traffic incidents. As such, the agency chose the I-270 as a test case due to its high traffic volume, high crash frequency, and continuously changing conditions. Eventually, the geographic area of the pilot encompassed the entirety of the St. Louis district, which covers every interstate and highway in the metro area. The project only covers interstate freeways - arterials and local roadways are not part of the scope.

As a high-construction area, lane and ramp closures would be constantly shifting over time. The difficulty of predicting crashes under these conditions with traditional methods made the deployment of advanced analytics and ML a natural choice. MoDOT hoped to use historical data to make better-informed decisions and minimize the chances of crashes and congestion in the project area.

Three technologies are covered as part of the project scope: predictive analytics, advanced video analytics, and weather analytics, or integrated modeling for road condition prediction (IMRCP). The project is funded as a 50/50 split between grant funding and self-funding, with the former going to deployment of predictive analytics and the latter toward the other two technologies. The entirety of the deployment was about 70 percent done when this case study was completed. Although these applications are being developed, full integration into the overarching ATMS is still ongoing.

Machine Learning Applications

The traditional MoDOT system for identifying and responding to incidents is highly manual. It involves between four and seven operators, depending on the shift and staff level, located in the St. Louis Traffic Management Center (TMC) monitoring over 600 traffic cameras in their district to identify, locate, and respond to incidents such as crashes. In addition to this monitoring, they rely on sources such as police or customer calls to provide awareness about incidents. Often, by the time the center is aware of an incident, downstream effects such as congestion or secondary crashes have already occurred. In addition to predicting areas of high crash likelihood, this project was meant to more quickly identify crashes that have already occurred.

The predictive analytics application of the I-270 project is the main module, as the other two provide inputs to allow it to function more effectively and process more variables. The application does not take automatic mitigative action but instead informs and gives recommendations to human operators, who then determine what actions to take. MoDOT plans to eventually be able to position its emergency response vehicles in areas of high potential crash risk to quickly respond to incidents, utilizing resources more

efficiently and cutting down response times for crashes. All three technologies have been in use in the live environment since January 2022.

The ML solutions used by MoDOT are sourced from proprietary vendors and therefore details about algorithms in some cases are trade secrets; however, high-level descriptions of the different technology modules can be found below.

• Predictive Analytics – The main platform, the predictive analytics processes real-time and historical data to quickly identify trends and conditions that could lead to crashes before they occur. It most directly interfaces with the state’s ATMS. The AI techniques utilized include pattern recognition and deep learning, with data fusion and edge processing techniques facilitating effective algorithm implementation. The roadway crash model combines predictive forecasting with deep learning AI to output a risk score for a roadway segment. The model is also able to provide explanatory features that explain the components that contributed to the risk score. The module is designed to formulate predictions up to 24 hours in the future, although it typically is used for more short-term predictions. MoDOT finished with development as of August 2021.
• Advanced Video Analytics – The video analytics module is integrated into the predictive analytics platform. It uses traffic cameras installed along freeways to detect incidents at specified locations. The algorithms rely on unsupervised learning, adaptive feature pruning, and boosting. For privacy protection, the processed images and videos are disposed of after being used in the algorithm. Development was completed as of January 2022, and it has been integrated into the predictive analytics application.
• IMRCP – The IMRCP weather forecasting and road condition prediction module was customized for use in the I-270 Predictive Layered Operation Initiatives project. It uses ML models to forecast future traffic speeds. Training for the off-line models is based on 2 years of historical weather, event, and traffic speed data on particular road network links, with monthly updates. Online modeling adjusts to real-time speed and incident data. The results provide near-term, 5-minute predictions of speed for each network link up to 2 hours into the future. Development was finished as of February 2022, and it is in the process of being integrated into the predictive analytics application.

Data Storage and Collection

Finding the correct volume, velocity, and types of data required to train an ML system is a common challenge in ML deployments. Originally, MoDOT only had access to vendor data acquired from Waze and HERE. After initial performance tests, they learned that their probe data did not report speeds on a road segment at the level of accuracy they required. As such, they decided to purchase supplemental data from additional vendors. Table 21 provides a list of data sources and the month when the data integrations went live (MoDOT 2022).

Table 21. Data sources and types.

Additional data sources that the agency hopes to integrate include the St. Louis County CAD system, Surfsight, and dash cam footage purchased from a vendor.

Data fusion proved to be a major challenge for MoDOT. In particular, the problem of duplicate incidents was impeding the project. Because multiple sources might provide alerts for the same incident and the system didn’t have any way to recognize and filter them, it would count duplicate incidents as separate. Waze presented a similar problem, in that multiple travelers might report the same incident, with the different reports counted separately.

The project team understood that the quantity of data vendors is not totally correlated with enhanced performance. Instead, MoDOT focused on identifying which data provide the best value to their particular use case. To this end, they are conducting evaluations of the various data sources to determine which ones contribute most to the analytics applications’ predictive power. The testing involves withholding certain sources from the model and observing how it impacts the accuracy of predictions. They are separating which data are worth investing in and which ones can be left aside with little negative impact.

Organization, Workforce, and Stakeholders

MoDOT did not have internal expertise and skills in AI/ML, therefore the bulk of the analytics work is being done by outside companies. Rekor is the main developer of the predictive analytics module, Traffic Vision provides the video analytics, and Synesis provides the weather analytics. Additionally, the agency hired a consultant to help with project management of the overall I-270 grant project. Although not having AI/ML expertise, the core team at the agency were champions of the technology throughout the project life cycle. They cited dedication, perseverance, and willingness to take risks as key factors that facilitated project success. Agencies often are comfortable sticking with operational methods with which they are familiar, so changing the normal course of business requires patience and tenacity.

The first step from an organizational perspective was to assemble a core team with varied perspectives. MoDOT formed a forward-looking team that included an engineer familiar with the AI/ML product and a TMC manager. Next, it was imperative to secure buy-in with the agency leadership. As the agency has multiple leaders who are open to new ideas, this was achieved early on.

A more daunting challenge was securing the buy-in of operators who would be using the ML applications on a day-to-day basis. Originally the team had two TMC and two emergency response operators test the program. Difficulties surfaced because operators were testing the new systems in addition to doing their regular jobs. The project team did their best to accommodate the workforce and avoid making them do double work, but it led to negative feedback early on. Additionally, the operators were not happy with the initial versions of the applications because of the low accuracy the system was demonstrating early in the training and tuning process.

There was a clear difference between leadership, who wanted to be forward-looking, and operators who were focused on completing their tasks during a particular shift. The project team learned the importance of managing extra burdens, facilitating communication of viewpoints between on-the-ground staff and leadership, and tempering expectations while explaining that ML deployments are iterative projects that will improve in accuracy over time.

Evaluation

MoDOT was most interested in identifying two key metrics: how fast the ML system can identify incidents compared to the normal procedure, and how accurate the predicted crash risk locations are. The latter was achieved by comparing existing crashes with the recommended crash risk area. When those two overlap, they are marked as a true positive. MoDOT found that even with this metric, it was difficult to

evaluate and more importantly communicate the results to operators. This is because the predictive analytics module is not designed to determine locations of incidents once they have occurred, but instead mark the locations that are at heightened risk of crash occurrence. Just because a crash did not occur does not mean that the system is malfunctioning if the conditions signifying high risk were present. Operators were skeptical of the results in these cases, claiming that the system was faulty when no crashes were present in the risk areas. Over time, the project team was able to help educate staff on the nature of risk scores.

Originally, the true positive rate was quite low. They estimate that out of one hundred crashes determined by operators, the ML application would predict one. With more data and tuning of the algorithm, the application has improved substantially.

Table 22 shows how the performance of the incident identification algorithm has changed and improved over time. It compares the rate of incidents that were first identified by the ML application to ones that were first identified by other, traditional means. If the ML can more quickly identify an incident, it is considered useful in an operational setting. We see that early on, traditional methods were still picking up most incidents, but as of the most recent evaluation, the ML application detected more. This speaks to the ability of ML systems to improve over time with continuous training and tuning. Additionally, we see that improvement is not always linear in ML, as the fraction of incidents first identified by ML decreased from November 2021 to February 2022. However, with additional time and attention spent on improving the algorithms, the fraction of incidents first identified by ML greatly pulled ahead of traditional methods by October 2022.

Table 23 shows the performance of the video analytics module for identifying crashes that have occurred. As this technology is already quite well developed, it started off with fairly high performance and has increased non-monotonically since then.

MoDOT has developed a winter weather event plan that will allow them to evaluate the IMRCP module after winter events for the season have occurred and as such, results should be available later in 2023.

Costs

The project has a total budget of $2 million, spread over four years. One million was provided by the grant funding and another million was matched by MoDOT. Technology costs ended up in line with initial expectations. Labor costs for consultants proved to be more difficult to estimate properly. This is because as the agency was integrating more data and functionality into the applications, hourly labor costs would rise proportional to the increased amount of reporting that they had to provide. Even with this unforeseen factor, they were able to stay under budget and therefore able to purchase supplemental probe data, improving system performance. Table 24 provides a breakdown of the estimated costs for a supplemental data purchasing agreement for the predictive analytics module.

Lessons Learned

Do not assume off-the-shelf options are always more time-saving than custom solutions

MoDOT picked commercial off-the-shelf solutions for their ML deployment believing that this approach would be a cost, labor, and time-saving decision. They had also spoken with vendors who offered to build ground-up solutions but expected that a commercial solution would be up and running faster. MoDOT did not anticipate the amount of work that would be required for integration between their systems and a solution that was not built specifically to cooperate with their architecture. Because they worked with different vendors for applications, those applications were not naturally built to coordinate with each other either. Once the ML applications were integrated with each other, integration of the systems into the agency’s ATMS was also onerous. Looking back, MoDOT would have more seriously considered a custom solution.

Promote awareness and set realistic expectations of AI/ML among staff

When implementing new technology, there will be varying levels of familiarity and knowledge among agency staff. Therefore, MoDOT AI/ML project staff wanted to set realistic expectations of the technology by explaining how it can be useful for well-defined problems. For example, operators may be discouraged by poor preliminary results if they are not aware of how ML works by improving over time as it is trained on new data. Additionally, deploying new technology is a sometimes lengthy process with unforeseen challenges. Setting realistic expectations for time to deploy prevents leadership from being surprised when delays in the deployment occur. Promoting awareness of the capabilities of AI/ML as well as deployment challenges throughout the organization before and during the project could have set more realistic expectations and resulted in a smoother deployment experience.

Have a plan to evaluate data types, and if useful, pare down data sources

MoDOT had many different types and sources of data; however, not all of it represents the same value to their AI applications. Some data might be less accurate or lower resolution than others, and some are just not useful for particular tasks. Additional data usually comes with additional costs, not just for purchasing or collection but also in processing, storage, and integration. It’s important that a team implementing an AI project have a plan to evaluate the usefulness of their data and if justified, cut some sources. MoDOT would leave certain data out and retrain their models to evaluate how impactful they were on system performance, thus determining which are worth investing in.

Design system outputs for maximum usefulness for the personnel who will work with the system

The predictive analytics application outputs locations that the algorithm has determined are at heightened risk of crash occurrence onto a map. The TMC operators use this map to inform their crash prevention and response strategies. The algorithm can predict up to 70 locations for high crash risk per 3-hour period. Although the system has this capability, the team understood that operators would not be able to meaningfully act with up to seventy locations on a map. The team configured the outputs to show only three of the locations at the highest risk so that operators could more easily utilize the predictions. After observing this configuration, they found that an insufficient percentage of crashes were visible to operators. The team then decided to show five high-risk locations on the map as a trade-off between usability and coverage. They were able to maximize the value of the ML system by tailoring the outputs of the system to the needs of the users and the organization’s operational goals.

Delaware Department of Transportation (DelDOT)

In 2019, The U.S. Department of Transportation’s Federal Highway Administration (FHWA) awarded a grant of $4.9 million to the Delaware Department of Transportation (DelDOT) as part of the Advanced Transportation and Congestion Management Technologies Deployment (ATCMTD) program for the Artificial Intelligence Integrated Transportation Management Systems (AI-ITMS) Program (FHWA 2019). This case study details the experience of DelDOT in managing this deployment, including their approaches to data and infrastructure, workforce and stakeholders, and evaluation, as well as challenges and lessons learned.

Background

More than ever, large, high-velocity traffic data, such as speed, congestion, traffic volume, and incidents are being generated by roadway sensors for use by Traffic Management Centers (TMCs). These data are collected via field devices, such as traffic detectors, CCTV cameras, and signal cabinets. Although some of this data is utilized for real-time decision-making or off-line planning, much of it often remains unused on servers. Advancements in data analytics and Machine Learning (ML) have shown to be capable of unlocking new capabilities in processing and analyzing big data, opening new possibilities for real-time decision-making at TMCs (Gettman 2019).

The DelDOT TMC Operations team found that it was overwhelming for human beings to monitor and make decisions for the thousands of sensors that provide their TMC traffic data every minute. Unpredictable congestion during summer months due to concurrent roadwork and tourism was of particular concern to the state (Vasudevan, Townsend, Schweikert, et al. 2020). They decided that ML would be a well-suited solution for their large-scale traffic data analysis needs and would help them understand traffic conditions and make recommendations for response plans. With the ATCMTD grant funding, they set out to design,

develop, and deploy the AI-ITMS program as part of the next generation of TSMO software systems for improving safety, efficiency, and air quality. The system was designed to address the following transportation problems:

• Multi-source traffic data fusion and traffic anomaly detection
• Short-term traffic flow prediction
• Proactive traffic management with live system performance assessment
• Simulation-assisted traffic response plan evaluation
• Live ATSPM and vehicle trajectory data for signalized corridor evaluation/optimization
• Multi-model decision support for traffic incident management
• Edge-computing machine vision for traffic monitoring and vehicle reidentification

As of January 2023, the following TSMO capabilities are deployment-ready: automatic incident detection and localization, short-term traffic flow prediction, machine vision for traffic monitoring, Live SPaT and ATSPM, and simulation-assisted response plan evaluation. The following capabilities are projected to be ready for deployment in the near-term: incident and congestion mitigation decision support system (DSS), loop signature matching for vehicle reidentification, and predictive traffic signal control and operation automation. Although some are deployment-ready in terms of development, they are marginally impacting operations as of this writing. Integration with state IT systems and their TACTICS signal control system is the final step before they embark on operational testing in live environments.

The AI deployment tests will be divided into several geographical areas with their own considerations and goals. These are the US-40 and I-95 interstate corridor, the US-13 and DE-1 corridor near the capital of Dover, and the beach area in the south. One example of a specific consideration is that the beach area experiences very high congestion when the tourist season starts in spring.

Machine Learning Applications

The AI-ITMS portfolio comprises many use cases over several geographical areas in the state of Delaware. These applications are at different stages of the development cycle and therefore differ in maturity. While some applications are deployment-ready, because of the extensive integration testing required for changes to operations, none have been put to control any devices as of this writing. Figure 9 outlines the framework with which DelDOT has approached AI application integration to TMC processes. Machine Learning provides opportunities to support traditional TMC operations at every step: data ingest, analysis, and decision-making.

The team tried to follow several general design principles when developing their applications. The software should be able to make decisions based on its data analysis, but also provide support for human decision-makers and allow for human review of decision logic. In anticipation of model drift over time, they have included capabilities for the tools to update themselves by retraining on new data either on a fixed schedule or based on observed performance degradation.

Below are descriptions of the various applications developed under AI-ITMS and their statuses.

• Multi-source traffic data fusion and traffic anomaly detection are among the most mature applications. The software has already been running for one year. This application ingests and fuses big data from a large number of sensors and identifies deviations from expected traffic patterns in real-time.
• Short-term traffic flow prediction is still under development but has already shown to have promising performance. It has demonstrated more accurate results than historical averages for speed and occupancy up to an hour in the future. They have been seeing the most tangible success in predicting traffic flows 15 minutes into the future. The next step is using the predicted demand to improve their traffic flow simulation models, comparing the predicted demand of roadway segments with their capacities, thereby being able to predict the probability of congestion on route links in the near future.
• Proactive traffic management, planned for deployment further out in the future, will build on the pipeline above for predictive incident mitigation. It will use the probability of congestion and incidents on roadways and detours to adjust signal timings or employ other TSMO operations along those links.
• Simulation-assisted plan evaluation will help to test proactive traffic management capabilities before moving to a live test environment.
• Multimodal decision support for traffic incident management includes detection of incidents through traffic anomalies or machine vision detection with CCTV cameras, involving freeway and arterial operations and signal timing adjustments.
• Edge-computing machine vision for traffic monitoring and vehicle reidentification is already quite mature. CCTV cameras equipped with machine vision capabilities can reidentify the same vehicle across different parts of the road network over time. This opens up possibilities for higher resolution evaluation of traffic flows and also allows the TMC to assist law enforcement in their activities. Another reidentification technology is loop signature matching, which while potentially having wide coverage, is less mature.

Data Storage and Collection

Before beginning development of their AI-ITMS solution, DelDOT already had a trove of traffic data to leverage for model training. These sources include traffic detectors, weather data, travel restrictions, probe vehicle data, and CCTV camera images and videos. After early prototyping of ML algorithms, they found that additional data sources would help the system achieve expected levels of performance. To accomplish this, they made enhancements to currently existing data collection capabilities or turned to third-party vendors. An example of the former was instrumenting data loggers to track vehicle dynamics. They were able to use this source as probe vehicle data. One third-party source they contacted was Wejo, which they used to enrich the visibility of traffic conditions data. Below are the strategies that DelDOT used to fulfill their program’s data needs.

Data Warehousing

For operational data, DelDOT decided to use an on-site, internal server solution for data storage, as opposed to a cloud-based solution. They were able to do this because they felt that they had sufficient resources in terms of funding and human capital to accomplish such a task. They have made major

investments into their own architecture, procuring 10 servers internally to handle storage and computing, with redundant instances running on production and test servers. The team decided that building and owning as much of the data infrastructure as possible was the best fit because it was too risky to depend on outside actors to keep their TSMO operations running. They never wanted to be in a position where they had to tell politicians or the general public “that there is nothing [they] can do when the cloud is down.”

The one area in which the team decided to use a cloud solution was for the purposes of data sharing. They acquired a Google Cloud platform instance for the purpose of providing external views of their data. Because they work with a variety of internal and external partners, they wanted to make it easy and seamless for collaborators to have access to and use their data.

The maturity of the development team meant that they were able to build modularity into the system. This allows them to handle modifications to data structures and new data sources, thereby easily fulfilling change requests. This decision to think about systems integrations and change processes required an upfront investment and commitment, as it is not something that is easily added post hoc.

Data Collection

Data collection and wrangling presented a variety of challenges that required resourcefulness and experience to tackle. Early on, the team found that data quality issues were impacting the quality of ML predictions. Sensors can fail and communications can break up before being received. They developed new iterations of their algorithms that were trained on missing, corrupted, or polluted data to ensure that it is sufficiently flexible to account for detector and communications failures. They found that to sufficiently tune deep learning (DL) algorithms, they had to work with a higher quantity and higher resolution data than the TMCs typically had worked with. Features like lane-by-lane traffic analysis or minute-by-minute data feeds would typically not be necessary for previous TMC operations but become very useful as inputs to ML algorithms. Latency of data was another issue identified. Depending on the sensor type, some data would have delays between 5-15 minutes. This bottleneck would restrict the speed at which the algorithms could provide their analysis, detracting from the advantage of real-time data processing that ML can provide. They were able to invest in improved data pipelines and cut latencies to around one minute. A lack of labeled data led the team to recruit interns and TMC staff to help gather and provide labels to footage from CCTV cameras that would power machine vision applications.

Organization, Workforce, and Stakeholders

As complex sociotechnical systems, successful deployment of AI applications requires subject matter knowledge and coordination across the enterprise. DelDOT’s program called for cross-departmental input including IT support, networking, computing hardware and software, traffic engineering, and operations. A key question for agency staff wanting to invest in technology upgrades is how to win leadership buy-in. The AI-ITMS team convinced leaders by highlighting the ROI of the program. Even though design, development, and enhanced data architecture have a cost, they pale in comparison to roadway capital investments. Below are some of the internal and external human factors approaches that DelDOT’s TMC operations team emphasized helped accomplish their goals.

Integrated Planning

Often, when a state or local agency receives grant funding for a project in emerging technologies, those deployments find themselves without ongoing support from the agency after the grant period has ended. DelDOT is aware of such potential outcomes and mitigates that concern by building their programs, including AI-ITSM, into the everyday business functions of the organization. The agency produces an integrated transportation management strategic plan every five years laying out the priorities and focuses of the organization. This promotes collaboration, cross-review, and sustainability across divisions and

programs. Capital improvement projects are reviewed by the TMC operations team and they include their own earmarks for sensors and hardware needs. Technology is treated as just as important as roadway infrastructure. This integrated organizational approach means that although they have launched the program through ATCMTD grant funding, the outcomes of the deployments will not be left isolated from the rest of the agency’s business functions.

Internal Expertise

DelDOT has an internal software development team that, in partnership with consultants, develop all the ML applications that are part of the AI-ITMS portfolio. They invested in this expertise due to various considerations. One such reason is to design the systems for scalability, resilience, and iterability. Without a mature development staff, the program team would not be able to guide development toward a product that will be stable and robust to change in the long term. Another reason is to promptly fulfill change requests. If development was strictly up to contractors, the team would have to continuously write requirements documents to make even minor modifications. The technical expertise of the DelDOT development team allows them to take matters into their own hands and act quickly. The program works with one consulting firm, who provide support on specific tasks such as particularly sophisticated modules.

Stakeholders and Partnerships

The AI-ITMS program does not only affect the work of the TMC operations department or even just the whole of DelDOT. The team must consider how it will work with actors as diverse as signal shop maintenance, emergency response, law enforcement, transportation planning, and university researchers. They have worked with the University of Delaware on complex ML algorithms, with the DelDOT Planning Department on machine vision applications, and with law enforcement for vehicle matching applications.

It is imperative that the whole operations staff understands the changes and advantages the AI-ITMS program will spur. To this end, the DelDOT TMC Operations team makes it a priority to involve technicians such as TMC operators and maintenance staff. Training of street-level maintenance staff has proved crucial, and the lower retention rate in these positions requiring retraining has proved to be a challenge. If these personnel lack knowledge and capabilities, the system performance will degrade over time as data coming from sensors becomes less reliable. Ongoing operations and maintenance (O&M) and staff capable of carrying it out are required to mitigate model degradation.

Coordination with vendors and state IT systems is another key to successful deployment. Interoperability with vendors of traffic signal systems has been a particularly important factor. The team has worked to ensure education, buy-in, and compatibility up and down their agency, as well as horizontally with external partners and stakeholders. This approach helps to achieve sustainability not just from a technology perspective, but also from a sociotechnical systems perspective.

Evaluation

As part of the ATCMTD grant program, DelDOT wrote an evaluation plan that outlines how to measure the impacts of the AI-ITMS program post-development. The general heuristic of the plan is to evaluate algorithmic level metrics in the short-term and then measure system-wide effects as deployment rolls out. Examples of the former would be predictive accuracy or mean squared error, while examples of the latter might be number of delays, average travel times, and traffic system throughput.

Early evaluation results for the AI-ITMS program have been promising. They use historical data from traditional operations to benchmark their results. The traffic incident management application detected 85 percent of incidents before DelDOT sent out email alerts using its current system. They have seen 90 percent accuracy in detecting all types of incidents. They are working toward achieving less than 3 percent error in

estimating traffic speeds. The volume of traffic on arterials has proven a challenge due to signal operations and as such, they have observed around 15 percent errors.

DelDOT is careful to consider qualitative markers of deployment success. A mitigation plan for unforeseen errors and extensive testing in isolated parts of the live network help to manage the liability of testing cutting-edge technologies like ML. Planning for and managing this risk is a key marker of stable deployment. Long-term acceptance by TMC staff, field operators, and roadway users is also an indicator that the team will continue to monitor.

Costs

Retrospectives on costs and cost drivers, as well as how they compare to expected values, are an important aspect of program evaluation. For the AI-ITMS program, the DelDOT team acquired 10 computer servers at about $200k per year in total material costs. There were additional costs for software tools, IT support, and installation labor that drove costs as well. They expect that variable costs will reduce over time as the expertise of the department increases, the technology continues to mature, and the integrations with the rest of the state network develop.

Lessons Learned

Below are some of the key lessons learned based on the feedback from DelDOT.

• Some capabilities require commitment from the onset of design: Some aspects of the AI system, such as modularity, require that the design and development teams consciously build those features into the architecture of the system from the inception of planning. Trying to retrofit those capabilities after the system has already been built is often arduous and sometimes even impossible.
• Investing in internal capabilities can help deployment success: DelDOT invested in its software development team that worked closely with consultants in building all AI applications. As such, they felt that they had enough expertise to manage an in-house server, as opposed to a cloud, solution for data and software. This commitment to internal capabilities means that the team can be agile in making modifications to the applications, with little friction. This agility helps to create an iterative approach to system design that improves performance over time.
• An organization with integrated decision-making is crucial for program sustainability: Working on an island will not lead to long-term adoption of cutting-edge technologies. The technology team at DelDOT reviews project proposals by the capital project department and vice versa. The structure of DelDOT assumes that technology is a crucial element of transportation and not just an add-on. The program team also works with TMC staff and field technicians to gauge their experience with new systems. These measures help to ensure that the program will have a lasting impact after the grant period has ended.
• Design your detector system with maintenance and fusion in mind: The AI-ITMS system relies on thousands of sensors that cover hundreds of zones. Making changes to that system can be a huge endeavor, especially when working with legacy technologies. Practitioners must understand the range and capabilities of the system as it is and what capabilities the system will need to power the AI algorithms because making those adjustments will be impossible without planning. Learn what the most critical data are to system performance and make sure those sources are robust. Retaining knowledge about sensors not just from TMC operators but also from field maintenance technicians is crucial to maintaining useful data streams and having a healthy system in the long term.
• Interfacing with existing software from vendors is a major challenge: Most vendors that DelDOT already used did not have systems in place for interfacing with ML applications. Because ML in transportation is still in the early stages in terms of adoption, all vendors have not included those

modules yet. APIs and data exchanges had to be developed to achieve compatibility, which was an extra burden outside of the development of the AI tools themselves.

• Be mindful of networking and security requirements: The state communications network already had many requirements from interface and cybersecurity standpoints. When developing ML applications, the program team had to be mindful that their software tools and modules comply with all existing security requirements. This proved extra difficult due to the relative lack of best practices around ML cybersecurity. When issues did arise, they addressed them 1-by-1 and with patience.
• ML solutions require ongoing O&M and training: ML models are time-dependent. Changes in transportation patterns, behaviors, or edge cases can cause the model to become less useful over time. Agencies looking to deploy ML solutions should have a plan for ongoing monitoring, maintenance and, if necessary, retraining of the system in order to stay timely.