Ben Duffy's adventures with Machines, Music and Movement

Click to unroll/roll all sections

Links

Short Bio

Roboticist, AI Researcher and Computer Engineer hybrid obsessed with using whatever is necessary to create truly intelligent machines for the benefit of humanity.

Co-founder at thing -> bearcover and MTank

Mad-scientist but also a jazz cat and an ultra-learner. His life began with a passion for music (Diploma in Classical Guitar, and a few bands playing electric guitar, piano and saxophone). And in 2022 began learning how to handstand which lead to a new passion for human movement (gymnastics, calisthenics, yoga, breakdance, other dance, capoiera, etc) and acrobatics.

My journey through technology:
  • Hobby Game Development (2014-2017)
  • Deep Learning (2014+)
  • Web Development (2015+): Every job I have had required it
  • Computer Vision (2015+)
  • Data Science and Engineering, NLP (2017-2019)
  • Reinforcement Learning (2017-2019)
  • Full-stack Robotics (2019+): hardware (actuators, sensors, mechatronics and rapid prototyping) and software (perception, control, SLAM, navigation, manipulation, integration and testing). I love this because everything I have learned previously coalesces and accumulates to be useful to some extent for this field.
  • Radar and Digital Signal Processing (2021+): Trying to sense breathing through walls... @Bearcover we build night-time nursing home robots that save time by checking halls and rooms (without entering) - means more time for care. Was a fun journey from FMCW -> SDRs -> UWB.

Formerly worked at Accenture as a Data Scientist working on projects which included:
Expense Anomaly Detection, Marketing Analytics, Pricing Optimisation, Recommender Systems, User modelling with Knowledge Base Graph Embeddings and resume to job matching.

Publications

Blogs

Reports/papers

Short timeline

  • 2022: Got obsessed with calisthenics -> handstands (~40 minutes a day for 100 days) which lead to a new passion for Movement (gymnastics, calisthenics, yoga, breakdance, other dance, capoiera, acrobatics, etc). Started counting calories more and never enjoyed food as much since I could eat whatever I wanted (within reason) and still lose weight. Lost at least 12kg (in November at the time of writing). Bearcover-wise we got pre-seed angel investment, hired 2 non-contract full-timers which makes us 3 founders, 3 full-timers, 3 contract part-timers. Moved into https://motionlab.berlin/ garage office.
  • 2021: It seems that after our 2nd pivot (3rd idea) since moving to Berlin in March 2020 we are getting more customer traction. We've changed our name to Bearcover, we received the EXIST grant, incorporated the company, hired many people to help us out for our summer deployments; paid pilots with 4 nursing homes this summer. Robots which see through walls with radar and inform nurses if there are any problems, all without bothering residents.
  • 2020: Moved to Berlin with the two other founders of MTank to go full-time into creating (thing). The garage/"3 guys in an apartment building ugly robots" vibe, the rollercoaster and insane learning experience of startups (software, hardware, recruitment, management, fundraising, business, sales, marketing, just talking to hundreds of people, pivots, the ups and the downs, etc) is just epic and exactly what I want from life.
  • 2019: Really ramping up now: Left Accenture in May 2019 to begin properly learning robotics and started working full-time on the extremely ambituous startup/goal (like many naive people have in the past) of creating cheap, general and robust home robots for everyone.
  • 2018: 'twas a good year. For MTank, Accenture, public speaking, finishing my Classical guitar diploma, three (yes, three) DL summer schools and many other things in the full timeline below.
  • 2017: Started as Data Scientist in Accenture at the Dock Innovation Centre in March.
  • 2016: Finished 4th year Computer Engineering at TCD. Founded MTank. Did a 3 month internship at Intuition as a Full Stack web developer. After that, 6 month "hack sabbatical" to upskill for a job in Data Science.
  • 2015: Got my first job (and best possible first job ever) as a full stack web developer intern working at Pointy.
  • 2014: Banged my head off a wall (till there was blood), and suddenly became good at and began to love programming lulz. Finished 2nd year General Engineering and specialised into Computer Engineering for the last 2 years.
  • 2013: Was lost, felt bad at many things, even programming was hard. Wasn't sure what I wanted, regretted not doing music in college.
  • 2012 and before: Birth -> Kindergarten -> Primary School -> Classical Guitar (age 7) -> Secondary school -> became good at math and music -> had a crisis dichotomy about which career path to choose out of both but "accidentally" chose engineering -> Started 1st Year General Engineering (in Trinity College Dublin)

Full timeline

  • 2022: Got obsessed with calisthenics -> handstands (~40 minutes a day for 100 days) which lead to a new passion for Movement (gymnastics, calisthenics, yoga, breakdance, other dance, capoiera, acrobatics, etc). Started counting calories more and never enjoyed food as much since I could eat whatever I wanted (within reason) and still lose weight. Lost at least 12kg (in November at the time of writing). Bearcover-wise we got pre-seed angel investment from 10+ investors, hired 2 non-contract full-timers which makes us 3 founders, 3 full-timers, 3 contract part-timers. Moved into https://motionlab.berlin/ garage office. Also was the hardest startup year so far, Awh man, I've lived multiple lives. Robot got more stable in 2022, but still took down the robot for a while to focus on the part of "seeing through walls with radar" (quite hard). Had multiple founder disputes in all directions between the three of us. Worried we wouldn't get funding, but we overcame it all and now we're stronger.
  • 2021: It seems that after our 2nd pivot (3rd idea) since moving to Berlin in March 2020 we are getting more customer traction. We've changed our name to Bearcover, we received the EXIST grant, incorporated the company, won the jury prize in the Make It Lean competition, hired many people to help us out for our summer deployments; paid pilots with 4 nursing homes this summer. Robots which see through walls with radar and inform nurses if there are any problems without bothering residents.
  • 2020: Moved to Berlin with the two other founders of MTank to go full-time into creating (thing). The garage/"3 guys in an apartment building ugly robots" vibes, the rollercoaster and insane learning experience of startups (software, hardware, recruitment, management, fundraising, business, sales, marketing, just talking to hundreds of people, pivots, the ups and the downs, the highs and lows etc) is just epic and exactly what I want from life. Involved with every part of what is needed from full-stack robotics i.e. hardware (actuators, sensors, mechatronics, rapid prototyping and manufacturing) and software (perception, control, SLAM, navigation, manipulation, integration and testing). Will release videos over the months. In June 2020 we pivoted from house robots to home care robots for the elderly. While we did get a lot of interest from partners we couldn't validate the idea fast enough with the German elderly in a pandemic. We then pivoted in December 2020 to nursing home robots after one of our partners described to us the many problems in his nursing home.
  • 2019: Released: Website: ai-distillery.io/. Gave a talk on "An Introduction to Reinforcement Learning" for the ODSC meetup. Organised the first MTank meetup in Stuttgart on "AI Distillery and the Journey from Cups to Consciousness". Went to probablistic.ai and RSS2019. Left my Data Science job in Accenture after 2 years to go "all in" into Robotics with MTank (our 2nd vision) and a stealth startup I'm founding. Exciting stuff! Spoke at an RPA conference and Ancestry.com meetup in September and Predict - Europe’s Leading Data Conference. Left my job in May 2019 to study robotics full-time and create our startup: thing.
  • 2018: Released Multi-Modal Methods. Gave my first 4 talks (1, 2, 3, and as one of the winners of a Computer Vision hackathon at 4). Was on my first podcast. Finished my Classical Guitar Diploma after 17 years of training. 3 summer schools (ISSonDL, DLRL2018 (summarised into 30 tweets by me here and slides here), DeepBayes) and my first conference (HLAI). Created a 4 hour Reinforcement Learning course at: RL course. Began working on two big projects within the MTank:
    • From Cups to Consciousness (c2c). Training embodied agents for multi-task learning through natural language instructions in 3D simulated houses and eventually the real world. On GitHub here: cups-rl using ai2thor.
    • AI Distillery. Automatically modelling and distilling knowledge within AI. In other words, summarising the research firehose coming from related fields within ML/DL/RL/AI/DS/CS/Stats using embeddings (doc2vec, word2vec), information retrieval and visualisation techniques. Check out some of our work at: ai-distillery.io
  • 2017: Accepted offer from Accenture (at the Dock) for a Data Scientist role in March. Skills/things learned:
    Machine Learning, Deep Learning, AI, Analytics, Visualisation, Data engineering, Research, API and full-stack Web Development, Business skills and a lot of domain knowledge for each specific project.
    Language/communication skills (speaking, listening, writing, reading, thinking, blog writing, general storytelling and presentation skills, UX/UI/Design Thinking skills and Agile/scrum).
    Released A Year in Computer Vision. Went to my first summer school at DeepLearn2017. First TV interview (see my YouTube or messy project gallery).
  • 2016: Finished 4th year Computer Engineering at TCD, learned SQL and did a module in Augmented Reality. Watched the cs231n videos. Finished my undergraduate thesis on "Video Understanding" for which a first-class honours was received. Convolutional Neural Networks were used for Object Detection, Scene Classification and I used a statistical method for Shot Transition Detection. Results were shown in a web interface for any YouTube video the user chooses.
    Founded MTank and increased the rate at which I could read and extract information from the abstracts of thousands of papers, while skimming a select few and properly reading an even smaller subset. Began learning how to play the tenor saxophone. Did a 3 month internship at Intuition as a Full Stack web developer. After that, I took 6 months off for a "hack sabbatical" of intense studying at home (coding, courses, meetups, hackathons, writing, researching) mainly for fun and to upskill for some kind of job in ML/AL/DL/DS.
  • 2015: Finished 3rd year Computer Engineering. Did more projects in Computer Graphics (C++ and OpenGL), game development (Java, Android and LibGDX) and Computer Vision with OpenCV. Check the messy project gallery for pictures and videos of most of these. Got my first job (and best possible first job ever) as a full stack web developer intern working at Pointy. I didn't know Python or JavaScript before starting and now both are my favourite languages. Created this website.
  • 2014: Finished 2nd year General Engineering and specialised into Computer Engineering for the last 2 years. Suddenly got much better by banging my head against a wall and started to absolutely love programming. Plenty of side projects, reading and learning in my spare time. Made a few games using DirectX and went deep into C++. Starting learning Java, Android and OpenGL.
  • 2013: Sulked for a while and regretted not choosing music which was my first passion. Finished first year General Engineering and started 2nd year. Chemistry, Physics, Mechanics, etc. Did two modules on programming. After the first I thought I was bad at it, but I tried harder and practiced and then started to like it.
  • 2012 and before: Birth -> Kindergarten -> Primary School -> Classical Guitar (age 7) -> Secondary school -> became good at math and music -> had a crisis dichotomy about which career path to choose out of both but "accidentally" chose engineering -> Started 1st Year General Engineering (in Trinity College Dublin)

Principles

  • My two favourite things to do are: Learning and Creating. In and out. Take and give. They also both naturally complement each other i.e. the more you create, the more you learn. The more you learn, the more you CAN create.
  • Lifelong and Continual Learning. Learning is the spice of life. Learning is how your brain grows. Learning can be one of the greatest providers of happiness. Learning keeps dopamine levels high and gives incredible satisfaction that can last a lifetime. Therefore, in my opinion Learning is the best drug.
    Transfer Learning i.e. it is easy to see how learning and practising ping pong might help your tennis skills but I'd push the concept as far as saying that practising ping pong will help your essay writing skills. Now I understand this is a push but I think learning anything has deep, indirect, hidden and countless connections with everything else you learn/process subconsciously and consciously but this might be hard to accept so I will throw the concept of Meta-Learning into the picture.
    Meta-Learning i.e. Learning how to learn. Once you learn how to learn well, you can learn anything. How do we get better at learning itself? By practicing learning constantly while simultaneously analysing and finding better ways to improve your own personal learning. Therefore, learning anything can help your ability to learn other unrelated things (Transfer Learning again and I think it makes my "ping pong -> essay writing" point at least somewhat true in this sense). It is sad that the state of education today often discourages learning more than it encourages it. #feelthelearn
  • Creating... To summarise my main passion in life: I like to build BIG things, the bigger, the greater, the better. So much else to say about this but I'll leave it at: very little can give more meaning and fullfilment than the act of creating something new directly from your own imagination. Learning is great and all, but only your creations will live on after you die i.e. they are your offering to the world and they carry on your legacy.
  • I try to make everything as simple as possible by using logic, breaking it into sub-parts, reorganising, focusing and using a first-principles approach. This helps me understand things to the very inner core and throughout my life this has helped me and proven to be very effective to everything from music to programming to reading to writing to understanding very complicated ideas etc.
  • Consistency and Focus is key. Great good can be accomplished through focus. It's not only how hard you work, it's also how smart and focused you work. One man's hour can accomplish what takes another man ten. Also on a related note, the 10,000 hour rule can be very discouraging to many people, you can get subjectively "excellent" at things within 100 hours (or at least get a "Woooow man, you're really good at that!").
  • Two factors to personal success: Belief and Hard Work (which is Smart, Focused and Consistent as well). Self-help books complicate this by making this list 100 points long. Also, of course "belief" often leads to motivation, passion, laser focus, etc.
  • Two factors to business success: A good idea and good execution of said idea. Everyone and their grandmother has 20 great ideas, good execution is much harder to find.
  • More

Programming Languages

Main used languages in order of preference/frequency used

  • Python - "Python gives you wings", "Python is like English, you tell the computer what you want" - Dogmatic Ben Duffy
  • JavaScript
  • C++
  • Java
  • C#
  • Matlab/Octave
  • SQL
  • Assembly (ARM and IA32) - back in college

Other "Programming Languages" used

  • HTML
  • LESS/CSS
  • JSON - "JSON is GOD" - Ben Duffy
  • XML (XPath, XSL), YAML

Fields/Specialisations

The below is 2-3+ years old but I'm not bothered updating it or deleting it.

Machine Learning/Data Science/AI

This is my main passion as this is currently how one can solve the world's toughest problems. I love to play around with data. And as I said above in my principles - All that there is, that a computer does/has, is a list of two things: Computation and Data. I want to master both.

Main fields/areas within AI

  • Robotics. This is my main focus now.
  • Deep Learning. I intensely follow around 2000+ papers a year in this and similar areas and I'm very up to date on the current state of the art of the field. I completed my Bachelor's thesis on using DL for "Video Understanding"
  • Computer Vision
  • Reinforcement Learning
  • Natural Language Processing
  • Generative Modelling and Unsupervised Learning

Other fields/areas that I'm interested in

  • Meta-Learning
  • Bayesian modelling and Graphical Models
  • Few-shot Learning, Semi-supervised Learning, Transfer Learning, Multi-task Learning, Continual/Lifelong/online learning
  • Hierarchical RL, Imitation Learning, Multi-agent RL, Inverse RL
  • Neural Program Induction/Synthesis
  • Domain Adaptation and Cross-modal/cross-domain Learning
  • Memory Augmented Learning
  • Reasoning/XAI/Common Sense/Physical Experiments/Graphs

General Algorithms

  • Linear Regression, Logistic Regression, SVMs, Random Forests/Decision Trees, K-means, K-nearest Neighbours, Matrix Factorisation, Naive Bayes, Dimensionality Reduction (PCA, t-SNE)

General Data Science skills

  • Probability/Statistics
  • Data Visualisation
  • Data Cleaning
  • Web Scraping
  • Research. Reading AI research papers and keeping up with all the latest developments in technology as a whole but with a focus on AI and all fields it is related to.

Deep Learning Frameworks

  • PyTorch
  • Keras
  • TensorFlow
  • Caffe
  • ConvNet JS

Other Frameworks/Libraries

  • NumPy
  • pandas
  • Scikit-Learn
  • matplotlib
  • seaborn
  • bokeh
  • D3.js
  • OpenCV
  • Jupyter/Ipython

Graphics, Visualisation and Game Programming

I have made games in Java, C++, Python and JavaScript using the below frameworks. I have used D3 for visualisation and generally just like to make pretty looking things.

Frameworks/Engines/Libraries

  • LibGDX
  • Phaser.js
  • DirectX
  • OpenGL + OpenGL ES
  • WebGL
  • Unity
  • pyglet

Web Development

Client Side Frameworks/Libraries

  • jQuery
  • Bootstrap
  • Templating with Handlebars, Jinja2
  • Angular JS (1 and 5)
  • Google Maps API
  • D3.js
  • Leaflet JS
  • WebGL
  • LESS

Server Side Frameworks/Libraries

  • NodeJS
  • Express
  • Google App Engine (using Python and webapp2)
  • Python Flask
  • Socket.io
  • Grunt
  • Azure PaaS
  • Consul

General Web skills

  • Security (SQL injection, Cross site scripting, TLS, etc)
  • HTTP + DNS
  • Microservices
  • Go Continuous Delivery
  • RabbitMQ
  • NoSQL (Cassandra, MongoDB, App Engine Datastore)
  • SQL (MySQL, T-SQL, Oracle SQL)
  • REST (documenting APIs, creating/debugging APIs, Postman, Fiddler, etc)

Other technical skills

Operating Systems

  • Windows
  • Linux
  • OSX

Other

  • Git
  • Android development
  • Agile
  • Design Patterns
  • Video editing, powerpoint, graphing/mindmap tools
  • Testing (Unit, Integration)
  • Microservices
  • Continous Delivery/integration
  • IDEs: PyCharm, Visual Studio, Sublime Text, Atom, Eclipse, Android Studio, WebStorm, IntelliJ, IDLE Python 3, Arduino, Keli uVision 4. Can sometimes, on a good day, exit Vim

Music

Instruments

  • Classical Guitar ()
  • Electric Guitar ()
  • Piano ()
  • Saxophone ()
  • can play a few notes in successive order on many other instruments...

Bands and projects

  • The Brickfield Park trio (Jazz/Funk mostly)
  • Panic at the Bustop (Busking acoustic guitar duo)
  • The Hacks (Alternative Rock)
  • The Scruffs (Alternative Rock/Rage Against the Machine covers)
  • SoundCloud

Other Interests

  • Generalisation
  • Specialisation (it doesn't have to be only one or the other)
  • Philosophy
  • Psychology
  • Engineering and Science in general
  • Technology applied to anything
  • Business/Economics/Finance
  • All five uses of language: Listening, Speaking, Reading, Writing and Thinking.
  • Neuroscience
  • Rockets and space
  • The Future
  • The Present
  • The Past
  • Listening to music
  • Travelling
  • Geography
  • Books
  • Films
  • Comedy
  • Cycling
  • Exercise
  • Almost everything else