Experience

Although I had lots of excellent exposure to the world of AI in my previous co-op, this term I was able to really dive deep into the fun of researching, data wrangling, model implementation and training. I got to touch all parts of the system and push different models until they broke.

Replicant is a California-based company that is remote-first. The company develops Thinking Machines™ which are conversational bots that are designed to handle customer calls and messages through natural language. Gone are the days of “Press 1 for Road Side assistance; Press 2 for Profile updates.”

Conversation for these bots can be distilled into four steps: converting the audio into interpretable text, understanding the user, decision making, and converting the response back into human understandable audio.

Most of the machine learning that my team and I worked on was trying to understand the user by extracting their intents (goals or desired actions) and the detailed substance of their statement — ie names, dates, products, etc which are known as entities.

I worked on three unique projects over my term. Each of them touched on a new area of the product: Make, Model and Year extraction, multi-lingual intent recognition, and custom entity extraction.

For the first project, I had to fine-tune a RoBERTa language model to accurately extract the make, model and year of a car (SUV, RV, truck, etc…). A RoBERTa model is a large pre-trained BERT model which uses transformers to perform incredibly accurate named entity recognition (NER). Specifically, they are trained to extract entities like organizations, persons, locations, dates, and misc from a sentence. In order to train this model to work for my needs, I compiled and cleaned a large car dataset from multiple sources with the makes and models of many of the cars on the road. Then I created several sentence templates, filled those templates with the data and ran the training on these fake sentences. When deployed to production my model had a >99% f1 score.

The second project I worked on was more of a research investigation – a “SPIKE”. There was a big external push from the customers and potential customers for our bots to support other languages. So our models needed to be able to extract intents and entities regardless of the language. The SPIKE I was looking into was determining how to make DateTime and number extraction work in French and Spanish. This required me to read several papers and analyze our code (pre-processing, training and inference) to determine several paths that we could take. Then, I created an implementation plan for those options. Finally, I developed a testing framework/guidelines to compare all of the options. This involved speaking with the sales team and the operations team to determine key business requirements (real-time extraction, accurate) and convert them into measurements compatible with machine learning (low latency inference, f1 score over 95%).

The final and most exciting project of my term was developing a model for custom entity extraction. The RoBERTa models that we use for other entity extraction tasks require large amounts of training data in order to achieve high accuracy. However, in several circumstances, there is little to no pre-existing data for the entities we are trying to extract. For example, if we are working with a health care company on sensitive information, there may not be any public data available for training. Or, in situations with industry-specific lingo or unique product types, the general models may not perform well enough. So in these cases, we need a way to extra these “custom” entities with little (Few-shot) or no (Zero-shot) training examples.

My job was to develop a model that Replicant could use for their own customers to create their own entities. Few-shot named entity recognition (FSNER) is a growing topic of research, however, it is new and not very collaborative; meaning there were many recent works, but they never used similar test sets, or improved on or compared themselves to other works. So my first step was to read, implement, and then compare several models that seemed promising. In order to compare the models for our specific needs, I curated two datasets to better measure what the models’ accuracy would be like in production. The first dataset was a merged dataset from all of the entity extraction models that were currently in use (make model, zip codes, pizza toppings, etc…). This would test how well the model could work on existing entities that we were trying to extract and would allow for a good benchmark to the models in production that are being trained on more data. The second dataset was a dataset specifically designed for few-shot training. Situations involving few-shot entity extraction are normally on really unique entity types so it is unrealistic to be training on entity types that are very broad (person, organization, etc…), and instead, it makes more sense to be training on more fine-grained types like (person-musician, misc-rivers, etc…). This dataset breaks those coarse-grained types into smaller types to better mimic the few-shot scenario.

The final step of this project was to use the knowledge I had learned and the models that I had tested to train a model for production. This involved doing extensive testing with different model types, configurations and tuning training parameters to ensure the highest f1 score. This training was all done on GPU machines on Azure. After the model was trained, I had to develop the interface to pre-process the data, run inference on the model, and post-process the response. And finally, I deployed it into production!

During the Winter 2021 term, I was fortunate to have the opportunity of working at an AI research division within TD Bank called Layer6 as an ML Engineering Intern. Layer6 had recently been acquired by the bank to satisfy their AI / ML needs. However, Layer6 was able to work very independently and on their own research, kind of like an internal contractor. This acquistion may have limited some of Layer6’s flexibility, but it gave them accesses to much larger resources in terms of data, money and talent.

My job within Layer6 was as an ML Engineering Intern. In this role, I was working alongside other developers to create core tools and explainable AI applications to help the Machine Learning Scientists (MLS) with the research, training and understanding of their models. For the term, I was tasked with creating a web application that allowed MLS to view the embedding of their data in an interactive, informative and engaging way. This would allow MLS to visualize how the model is interpreting the data and compare it to their expectations. The application allowed MLS to filter their data, and then view the model’s embedding of that data in an interactive 2D latent space. They could then zoom, pan and click through this embedding space to explore their data. There were additional settings to control the number of clusters, data points, dimensionality reduction settings, etc…

Once the first version of the tool was completed, I got feedback from the MLS. This feedback occurred through eight cognitive walkthroughs in early April. From these interviews, I uncovered that there was a need to have a further understanding of the underlying data and how points relate to one another. Clicking each point and parsing through the information available in the preview area was unintuitive and not as immersive as they had hoped; they needed a way to view the instance’s data within the visualization. This inspired my interviews and a survey for glyph-based feature vector visualization research.

There were 5 major steps involved in the creation of the visualization tool I created: Data collection, embedding extraction, dimensionality reduction, clustering and the visual interface.

1. Data Collection and Inference

The first step involved gathering all of the necessary data from an ElasticSearch instance. This also involved constructing the complicated queries required to fetch the data under several filtering parameters set by the user. These filters included, the data set (training or test), the data range, feature value range, prediction percentile or even a combination of these criteria. Once the data was collected it was passed on into the model to obtain an embedding.

2. Embedding Extraction

The embedding of the model is a compressed or encoded version of the input data. In neural networks, a hidden middle layer could be used as an embedding. Although, at Layer6 all the models use a gradient boosted random forest structure called
XGBoost. An MLS at Layer6 developed a technique that leverages the physical tree structure of the model and truncated single vector decomposition to create a data embedding for XGBoost models. I first performed this embedding extraction technique to find a data embedding for the visualization tool.

3. Dimensionality Reduction

This embedding technique was able to reduce the dimensions of the input data from over 250 dimensions to 20-50 dimensions. However, to be interpretable to the MLS using the tool the embedding had to be reduced further to two dimensions. To achieve this reduction, I performed the MLS’s choice of t-SNE or UMAP. These two techniques are well established in the industry and use manifold learning to create a non-deterministic mapping of the data from higher dimensions into 2- or 3-D.

4. Clustering

The dimensionality reduction technique naturally reveals clusters based on the geographical layout of the points on the screen but in order to provide an additional layer of information I chose to overlay colour-based cluster information on the data points. The raw data was clustered using the hierarchical clustering technique using the feature values or the SHAP values. The SHAP values are values given to each feature of every instance between 0 and 1 that denotes that feature’s contribution or importance toward that instance’s prediction. Clustering based on SHAP values is a common explainability technique and leads to improved clustering because all the features are mapped to the same unitless feature space. After the clustering, each cluster is assigned to a colour and then the points are coloured accordingly. This added layer of information can help MLS check that the clusters that come from the model embedding align with the clusters formed through colour as a kind of second check. Additionally, the MLS can overlay clustering based on a different feature to verify or reject certain beliefs they have about the model’s interpretation of the data.

5. Visual Interface

After this dimensionality reduction is performed, the data is then sent to the front end where it is displayed using the D3 graphing library. In order to enhance the interactivity, I developed a way to dynamically display the points stored in a k-d tree on the screen as the user pans and zooms. There was also additional information about a cluster or a point available in a preview area to the right of the plot when a point was clicked on by the user.

This past summer (Spring 2020) I got the privilege of working for TD at their Student Innovation Lab. This term started off a little wild and had many struggles pertaining to COVID and remote work, but I am glad how everything turned out in the end.

The spring started with a whirlwind of co-op confusion. I had received a couple of offers at the end of WaterlooWorks’ main round and was excited to have some options about how to spend my next four months. After a bunch of deliberation, I decided to pursue an international co-op experience in Bonn, Germany with Elastic.io. Only a short while after accepting my offer COVID-19 devastated the world and left me without a job. I had to quickly scramble to try and find something. I reached out to TD, who had offered me a position in the main round, to see if they would take me on board for the summer. They had already filled all of their co-op positions for the term but thankfully and luckily for me they were willing to make an exception.

Now, with a job lined up, I finished up the school term online with several painful 4-hour exams. During this exam cycle, I got the unfortunate news that TD would be pushing the co-op start date and I wouldn’t be able to start work until June 1st. So, once my exams wrapped up I had 6 weeks to stay at home and do whatever I wanted. I tried to fill my time with productive work but in the end, I was most proud of the catching up I did on my Netflix List. I also got a cute new puppy named Mezzo in March so I spent lots of time cuddling and training her. During this time I did manage to update my web and mail servers and I started to work on some Arduino projects. I got started on developing a sliding doggy door that would open up with a Google Assistant command or a dog proximity sensor and I started to work with the UWaterloo Midnight Sun Solar Car Club.

Once work started remote on the 1st of June I was quickly thrown into the thick of things. The Lab, located in Communitech, is TD’s small Digital Solutions branch that focuses on developing banking solutions for their student clientele: “For students by students”. The team was comprised of 7 interns, 7 full-time workers (both dev & design), and one manger. It is a tradition at the Lab to start and end each work term with a Hackathon. Although I am not able to say what we did for these projects, I can definitely say they were the most exciting and memorable parts of the term. I was able to get experience with my first high-pace, high-pressure hackathon atmosphere and get closer to the other co-ops.

Although remote work provided many challenges and frustrations, the Lab did a good job of trying to keep us connected with many no-work meetings filled with Skribl.io, weekly coffee chats and daily check-ins. For most of the term, I was working closely with my mentor and another full-timer on the Lab’s top-secret financial app. We had to redevelop large parts of the code to compile with TD’s risk appetite, developed new features to enhance the user experience and developed better ways to aggregate data from multiple sources.

Over the course of the term, I was able to greatly improve my Flutter and dart skills and I was able to make some strong connections with the colleagues I was working with. The experience I gained during this co-op has already bore fruit as I was recruited by a local startup called Social Kindred to be their lead mobile developer in Flutter. The term ended with a challenge to design the traditional Lab term t-shirt. I choose to embrace the Work-From-Home theme of the summer and designed a minimalistic logo that won the competition. I look forward to getting the shirts in a couple of weeks!

I’ve witnessed all of the hype and demand for the Dark Theme grow over the last year. All major apps and mobile OSs have added dark theme support. You can almost get through a whole day in a fully-functioning dark theme world if you wish.

So I thought I would try it out for myself by designing a one-page, dark themed resume. My hope for this resume is to try and embrace the love for the dark theme and cut through some of the noise during resume screenings. I spent even more time trying to distill my work experience to one page and a minimalist design than I did on the design. My main focus for this design is to keep it short, concise and beautiful. I am hoping to try this resume out this term on large organizations with hundreds of applicants and see if I can get some interviews.

I would love to know what you guys think or tips for improvement so leave a comment below!

My second co-op term was at the Public Bike Systems Company (PBSC for short) located just outside Montreal, Quebec working as a Java Developer. PBSC is not a household name by any means but they are an international company that provides Bike-Share systems to clients (33 cities and counting) all around the world. We supply the software, docking stations, and bikes to popular ride shares like Bixi (Montreal), Toronto Bike Share, CitiBike (NYC), Careem (UAE), Bicing (Barcelona), etc…

The story for this coop started long before I arrived at work on the first day. After a smooth first work placement with regards to arranging my own job and finding housing with family friends. It came as somewhat of a shock as to how challenging it was to find a job through the University’s in house job search system, WaterlooWorks. Ending up at PBSC seemed to be less about my skills or resume but rather about my luck. I had applied to just under 50 jobs in the first round and only got two interviews… From there I only got ranked for one of my interviews. Yet, thankfully the person with the offer to PBSC declined and I was given this great opportunity.

This work term had slightly less stress and tempo than my previous placement. However, within my first week, I was in meetings to develop gamification for our system. One of the cities we work with wanted their users to be able to engage in incentives where they could gain redeemable points for doing tasks like redistributing bikes from populated stations to empty ones. My job for this feature was to configure how the prices would be assigned to users. I had to determine how, once the customer got their points, they would be able to store their prizes. I also created a system that would allow for configurable prize definitions so that a user could achieve customizable awards instead of just a default 30 minutes free voucher. These tasks involved extensive work with the database and creating controllers/facades in the backend.

After the development of this feature was complete, a lot of testing and bug fixing was soon to follow. In addition to the changes for the gamification feature, we need to keep up with the incoming bugs form other cities. That work consumed most of our time but with the time I had left, I started to dig into Courgette Automation Testing. The team had previously done work in this software but it was hardly developed and not properly configured to achieve its purpose. The automated testing was meant to eliminate the need for any manual QA testing in the frontend but it wasn’t currently being used by any of the QAs. I independently took on the task of building out this tool. I refactored and organized what had been done, researched enhancements and features within courgette to better leverage the system and finally expanded the coverage of our test sweets. I have now effectively reduced the manual testing time for our QAs by 95%.

Despite all the work that I did at PBSC there was a strong emphasis on work/life balance. We frequently took breaks to play foosball or go out for team lunches. Over the course of the term, I went from an absolute newbie at foosball to being the top dog in the office. I like to say that my most improved skill over the term, beyond any technical improvement, was my foosball finesse.

As the term comes to an end, I fondly look back on an internship of learning and wonderful experiences, and I look forward to starting the search for my next internship.

For my first work term as part of my undergraduate co-op program, I spent four months working in Winnipeg, Manitoba at The Wawanesa Insurance Company on the Web Content Management Team as an Application Developer. Wawanesa is the largest mutual insurance company in Canada and has recently been growing in California, Oregon, and Las Vegas. They are a company with a vibrant history on the west coast founded in strong community morals.

‘I learned so much about so much and experienced so much more’

Since this was my first real work experience it came with a lot of learning, surprising bureaucracy and firsts. I was quick to realize that despite having had four coding classes, I have never really gone through the processes required for coding in the real world. I learned how to use source control, set up environments, interact with servers, run builds on Jenkins, manage tickets and task flow in Jira, add documentation, etc. . . None of these tasks were particularly difficult or cumbersome on their own but when trying to learn all the particular lingo, logins and corporate-specific workflows all while trying not to look like a fool, it gets tiring. My advice for anyone going into their first workplace, or for that matter any new workplace, is to make friends with a junior developer on your team that you can go to for those small reminders you need. This way you can avoid inconveniencing your boss/supervisor all the time and start to develop other relationships within your team. In addition, this junior developer is the most likely to have recently gone through this new arrival situation.

As the term progressed I got more comfortable with my team and started to take on more challenging tasks. I worked on a large initiative called the Anual Mileage Questionnaire (AMQ) that required me to create a feature that would send the necessary customers an email with a link to our website. Once on the website, they would be able to update the mileage for all the cars on their policy. This took a lot of organization, coordination, and code. After this, I worked on adding a payment feature to the website that would utilize Morgan Chase Paymentech Portal to allow users to pay their insurance bills online.

Finally, in the last few weeks of my term, I got very involved in the company’s content management system, dotCMS. This software was very neglected and misunderstood by the organization. It was a tool that developers tried to dismiss quickly after they arrived at the company but also a tool that analysts didn’t quite understand. As a result, the usage and structure of the content was in disarray. I decided to take the initiative to try and sort out the mess and get a proper grasp of the tool. Using my growing unique skill set with this tool I developed the french website for Wawanesa Life. Before I left I made sure to document my work and give multiple presentations to both business people and developers about how dotCMS should be optimally leveraged and maintained going forward.

However, there was more than just technical experience that came along with this placement. I was able to foster close friendships and working relationships with many of my diverse colleagues. As a team, we represented 11 different home countries and surprisingly, I was the only member of the team born in Canada. Luckily, I learned a lot about other parts of the world, unique experiences and stories, and also gain a greater appreciation for the great country that I live in. In addition, I gained insight into various bureaucratic/corporate procedures. My department went through a very rough operational crisis. In this time many people were fired, safety mechanisms were put in place, and teams were rearranged. I got shuffled through four different teams, 3 supervisors and 2 directors. This helped me learn how to get my feet settled quickly in order to stay productive amidst the turmoil.

Overall this experience in Winnipeg was wonderful. I learned so much about so much and experienced so much more 🙂

I learned very quickly that in order to gain trust and repour with the campers I primarily had to be attentive and open to their needs: they are not all the same and can not be painted with the same brush stroke but rather need individual care and attention. This was exceptionally challenging when my Inclusion campers found it hard to express themselves or physically couldn’t speak their needs. In these cases patience and quick problem solving was the only way forward.

At Fraser Lake, I was also able to interact and create lasting friendships with the staff. The atmosphere of Fraser Lake – serene, inclusive, beautiful, peaceful and loving – is so conducive to fostering connections. In this place, I felt free to be myself and be at one with the present away from all the flurry of school and work.

The experiences I had at Fraser fostered a deeper sense of empathy and diversified communicative skills. I learned valuable lessons in patience and integrity, and found my passion for leadership.