HQ BARCELONA - Senior Data Engineer

Jobandtalent Who we are: We are a workforce on-demand company that heavily relies on tech. We integrate the complete process of selecting, hiring and managing employees, being the first in digitizing all the steps, introducing Artificial Intelligence solutions in multiple phases, obtaining faster and higher-quality service for our clients and great satisfaction and loyalty for our workers. Our Impact on Society: Our vision is to create a long-term job security condition for our workers while providing flexible solutions to our clients. We can do this by concatenating different short-term contracts and minimizing–or completely removing–their unemployed time. We want to remove the need for finding a job ever again.  Data-Driven: Being the owner of the entire hiring funnel, workforce management, and working experience, give us very unique data that we transform in powerful knowledge to make better decisions, improve job-candidate matches, estimate quality and affinity, and share performance feedback with our workers and clients. We are the most advanced workforce on-demand company thanks to our data and the way we use it. Great Challenges for the Data Team: Integrating very different AI solutions to improve the experience for both our workers and employers (Shift Optimization, Machine Learning algorithms, Quality Score Estimations, Document recognition, etc.), relying on a very powerful ETL that permits endless automatizations while keeping the focus on data quality. Growth Overview: An annual growth of 170%, available in multiple countries: the UK, Spain, Germany, Sweden, Mexico, Colombia, France, and more are coming. HQ in Madrid with a second big presence in Barcelona and some remote workers in the UK, The Netherlands, and the USA. And finally, some of our great clients are Amazon, XPOLogistics, Cabify, Santander, Just Eat, etc.  You will be the main referent of our data architecture, leading the design and implementation decisions of our data pipelines.  The Data Team is very independent of the rest of the Tech and Product teams. We build and manage our services that run our AI models as well as our data storages to quickly process the data we need, and let the Data Scientist performing the “Time Travel” to properly train our algorithms. A quick insight about our architecture and tools: We are building our new data lake and data warehouse directly in AWS S3, synchronizing the data via DMS, and consolidate it via Hudi. This data will be accessible by multiple Redshift clusters via Spectrum that allows us to quickly scale up across all the countries. In addition, as data is stored in S3 this could be exploited via Athena or an ad-hoc Spark job.  We have multiple queue systems from which we consume data coming from the Jobandtalent Message brokers while filling up our data storage.  We use Airflow as our ETL which allows us to break down any complex problem/process into smaller ones. Each component can be implemented dir...