Currently, speech recognition by smartwatches is implemented primarily through audio and voice recognition. This approach is not feasible in situations where users must vocalize privacy-sensitive information or in environments with background noise clutter. To solve this, silent speech recognition methods using RGB Cameras, Electrical Conductivity, Pyroelectric effect, and Optical Proximity Sensors have been utilized to model face and lip movement to recognize words. However, these methods are prone to ambient factors such as lighting, background, and skin tone. The goal of this project is to use depth-sensing as a silent speech recognition technique to visualize lip movement and recognize commands. My goal in the project was to create an interactive website that animates the depth data to visualize lip movement. Depth data was collected in Point cloud format using the True-Depth camera on an iPhone 12 mini and deep learning models such as the YOLO model, and RGB-based visual speech recognition model, AV-HuBERT, were used to correlate the depth maps with phonemes and visemes to recognize commands. Results indicated the system could recognize two pretrained command sets with sizes of 27 and 10 at 82.24% and 85.74% accuracy. I created a website to visualize the lip movement using Django, a Python-based backend framework, and a React frontend which utilized the Three.js library to handle animating the Point cloud data. The lip animations for 10 digits and 27 commands(the corpus of data used to train the deep-learning model) from five different speakers can be selected and viewed by the website user.

Nitrogen vacancy (NV) centers in diamond have emerged as one of the leading solid-state quantum systems. Their optically addressable spin states make the NV system appealing for magnetic field sensing and imaging applications in condensed matter physics, neuroscience, geophysics, and device analysis. However, the low absorption cross section of NV centers limit the conversion of excitation optical power to spin-dependent fluorescent signals, leading to high optical power consumption, therefore precluding applications where phototoxicity and heating effects are of concern. This issue can be addressed with diamond light-trapping waveguide structures, which have shown to increase optical path length via total internal reflection. Using a ray tracing simulation tool, I optimized tunable parameters such as beam incidence angle, beam incidence position, and facet geometry of a diamond waveguide to maximize optical path length. The results show an optical path length of up to 90 cm in a 3mm x 3mm sample. This represents an enhancement factor of nearly 300 in pump-to-signal photon conversion efficiency compared to a single pass geometry. The coupling of a green excitation beam into the waveguide structure is demonstrated with a precision experimental apparatus that allows translational and rotational degrees of freedom. I show that the mechanical polishing of a diamond waveguide minimizes scattering losses and enables us to achieve the computationally predicted optical path lengths. Its potential to address a greater number of NVs within a given excitation power budget demonstrates a promising future for a compact and portable precision diamond sensing platform.

Micro-mobility vehicles such as e-scooters and e-bikes are becoming increasingly prevalent, especially in urban areas where traditional forms of transportation like cars are inconvenient. Our system aims to (a) generate insightful conclusions on the real-time condition of roads and sidewalks by using electric scooters and (b) discover which modes of data (audio, visual, sensor) gathered from the scooter are most helpful to this goal. A mobile app continuously draws IMU (Inertial Measurement Unit), GPS, photo, and microphone data from a smartphone mounted on the electric scooter. Throughout a user’s ride, data is uploaded to and accessible from a cloud storage platform. Images are classified into road type (asphalt, brick, sidewalk) and road quality (smooth, worn, ruined) through two separate convolutional neural networks (CNNs). A third CNN extracts road quality from the audio recordings. Acceleration (IMU) time-series data determines road quality via a supervised learning model. Road classification based on images offered the greatest accuracy (85-90%), making image data the most informative regarding road conditions, followed by audio data (likely due to intangible factors such as wind and voices), and lastly, acceleration (IMU) data. Users on the system’s website select two coordinates on a map and receive data on the various road types and qualities along the route between the coordinates, aggregated from all modes of data. Applications of this system include aiding city infrastructure planners in prioritizing which streets to renovate, producing more convenient routes for wheelchair-enabled people by considering road slope, and helping users maximize their comfort while traveling.

Field Programmable Gate Arrays (FPGAs) are a powerful alternative to CPUs and custom microchips for running computational workloads. While High Level Synthesis (HLS) is the mainstream programming approach for FPGAs, OverGen is an overlay architecture for FPGAs that has proven to be highly competitive with HLS-based designs. Overgen contains functional units (FUs) implemented with FPGA Digital Signal Processing (DSP) resources, which handle the arithmetic and logic operations of the processing elements. However these FUs are not implemented in the most hardware efficient way - different operations such as multiplication and addition are implemented on separate DSP slices, and their results are multiplexed. This not only increases DSP usage, but also power consumption. The purpose of this research is to fuse the arithmetic and logic operations by using the same DSP slices for different purposes, thus saving in FPGA resources for each FU. The OverGen overlay generator would then be able to map more processing elements onto an FPGA than before, and possibly achieve better performance on workloads. Furthermore, this research is a stepping stone to look at more generalized ways of mapping desired operations such as multiply-accumulate onto arbitrary DSP networks, to allow the overlay generator to use the DSP slices more efficiently. While the implementation in this research is a custom hardware fusion, perhaps fully exposing the capabilities of the DSP and the network that connects them to the software compiler could yield even better results.

Rapid detection of the toxicant HF is essential to the safety of emergency personnel combating fires. HF is produced in electric vehicle and battery fires, and in structural fires in modern developments containing synthetic polymers. We leverage the capability of in-situ infrared laser spectroscopy to detect and analyze HF production structure resulting from PTFE combustion with oxygen, a previously unexplored area of research. A laser targeting absorption features of HF and H2O, and a laser targeting those of CO, are pitched through the active combustion zone. Measurements are obtained at varied line-of-sight positions and over a range of fuel grain lengths to resolve 2D evolution of gas speciation. We assume an axisymmetric and quasi-steady state combustion process allowing our results to be Abel-transformed into the radial domain. As predicted by chemical kinetic simulations, we detected consistently high concentrations of HF in consecutive combustion tests, confirming repeatability of our detection method. We observe radial diffusion of HF from reaction layer growth and turbulent mixing as fuel grain lengths increase. Our spatially resolved, granular results demonstrate the robustness of the developed HF sensor and can help anchor PTFE combustion chemical kinetic mechanisms to improve predictions of hazardous HF production. Future research will include sensing and analysis of additional combustion-related toxicants, and development of a prototypical real-time detection device for emergency personnel. This technique can also be used to granularly and quantitatively analyze virtually any axisymmetric combustion, including those with important implications for hybrid rocketry.