Episode 334: In the Dynamics Corner Chair: Business Central Experience - Security, Scalability, and Updates

WEBVTT

00:00:00.700 --> 00:00:06.793
Welcome everyone to another episode of Dynamics Corner, the podcast where we dive deep into all things Microsoft Dynamics.

00:00:06.793 --> 00:00:16.952
Whether you're a seasoned expert or just starting your journey into the world of Dynamics 365, this is your place to gain insights, learn new tricks and how it all works in the back end.

00:00:16.952 --> 00:00:17.984
I'm your co-host.

00:00:18.024 --> 00:00:19.582
Chris, and this is Brad.

00:00:19.582 --> 00:00:23.393
This episode was recorded on July 26th 2024.

00:00:23.393 --> 00:00:27.550
Chris, chris, chris, what an amazing discussion.

00:00:27.550 --> 00:00:35.100
My mind is overflowing with all of the information we learned Today.

00:00:35.100 --> 00:00:47.914
We had the opportunity to learn more about the Business Central performance, online backup, security, performance and partner tools, and much more.

00:00:47.914 --> 00:00:50.981
With us today, we have the opportunity to speak with Dmitry Chaydev.

00:00:50.981 --> 00:01:08.311
Good morning, gentlemen.

00:01:09.512 --> 00:01:10.953
Good morning, good afternoon, how are?

00:01:10.972 --> 00:01:11.313
you doing.

00:01:12.373 --> 00:01:14.555
Yes, good time of the day yes.

00:01:15.156 --> 00:01:16.897
Excellent time of the day and it's Friday.

00:01:16.977 --> 00:01:18.918
Nice background bread there.

00:01:24.480 --> 00:01:26.463
Yeah, you know, chris has been sitting outside the past couple discussions that we've had.

00:01:26.463 --> 00:01:31.834
So I figured today, since I'm, you know, in the northern part of the country, I might as well sit outside and appreciate it as well.

00:01:31.834 --> 00:01:33.340
It's finally a nice day out.

00:01:34.081 --> 00:01:36.683
Fantastic Nice no great to see you, great to see you.

00:01:36.683 --> 00:01:37.364
Yes, nice to see you.

00:01:37.364 --> 00:01:38.224
The weather is great.

00:01:38.243 --> 00:01:38.664
Yes, yes, yes.

00:01:38.664 --> 00:01:40.084
How was your vacation Perfect.

00:01:40.084 --> 00:01:44.487
Yes, yes, yes, how was your vacation Perfect perfect.

00:01:45.429 --> 00:01:53.013
I spent the whole three weeks out there in Spain, with some of that time with my parents.

00:01:53.013 --> 00:01:55.295
So incredible to meet everybody.

00:01:55.295 --> 00:02:04.804
My brother lives there in Spain so we had some really good time there together and it's a nice area I can definitely recommend.

00:02:04.804 --> 00:02:07.370
It's not too far from Valencia, so the coastline is beautiful.

00:02:07.370 --> 00:02:12.188
Small towns along the coastline, nice mountains.

00:02:12.188 --> 00:02:16.963
For hiking I mean great area, so I could most definitely recommend that.

00:02:17.305 --> 00:02:41.296
I'm jealous and one thing it's very, very few people here in the United States take vacations like they do in Europe, and I wish we did, because that's one thing I'm envious of is that you know, in Europe, most of the countries, most of the individuals that I speak with take the vacations and enjoy the vacations and appreciate the vacations like they should.

00:02:41.296 --> 00:02:43.828
You know, doing things, spending time with your family.

00:02:43.828 --> 00:02:46.325
I know here in the United States very few do it.

00:02:46.325 --> 00:02:54.674
We have that you know we'll take a Friday off and have a three-day weekend, which really doesn't do anything, because I think with a three-week vacation

00:02:55.341 --> 00:03:00.272
you get to spend a couple days unwinding, you know, forgetting about work, kind of de-stressing.

00:03:00.272 --> 00:03:06.923
Then you get to have your vacation and enjoy it and then you know you have what I call, like you know, the sunday horrors.

00:03:06.923 --> 00:03:12.147
I guess you could call it, because then, a couple days before you go back, you start getting okay, what am I going to go back to, what do I have to deal with?

00:03:12.147 --> 00:03:25.611
So having that three week period of time gives you that, you know, chance to really detach and relax, whereas you know, if you do it for a long weekend or even sometimes just a week, you really can't let go.

00:03:25.812 --> 00:03:27.604
So I'm glad you're absolutely right.

00:03:27.604 --> 00:03:32.944
Like, practically I only had one week of vacation because you're absolutely right in the mindset.

00:03:32.944 --> 00:03:41.467
The first week was all wind world of what's going on in the office where I am, you know logistics and still catching up with a few things.

00:03:41.467 --> 00:03:47.181
But then the second week, full relaxation, full disconnect, no emails, no teams messages.

00:03:47.181 --> 00:03:56.332
And then the the third week was the time when I actually started thinking about work a little bit like, okay, I'm starting to get there.

00:03:56.332 --> 00:04:00.810
What's cooking there in the office, what's going on there, what's the next step there?

00:04:00.810 --> 00:04:06.248
So yeah, so it's a good timeline, absolutely but yeah I thoroughly enjoyed it.

00:04:06.500 --> 00:04:07.704
Yeah, really really beautiful place.

00:04:08.306 --> 00:04:10.050
I'm glad that you had a great vacation.

00:04:10.050 --> 00:04:15.031
Looking forward to speaking with you about Business Central online.

00:04:15.031 --> 00:04:17.286
You know server performance, upgrades and security.

00:04:17.286 --> 00:04:21.382
But before we do, would you mind telling everyone a little bit about yourself?

00:04:22.865 --> 00:04:24.148
Yeah, yeah, absolutely so.

00:04:24.148 --> 00:04:25.951
My name is Dmitry Chudaev.

00:04:25.951 --> 00:04:29.048
I'm a group product manager in Business Central team.

00:04:29.048 --> 00:04:46.759
I live here in Copenhagen with my family for quite some time for more than 20 years now and I'm embarrassed to mention that because all of my Danish colleagues they start asking me why don't I speak Danish to them, and, yeah, that is embarrassing.

00:04:46.759 --> 00:04:58.427
I do understand Danish, I do speak Danish, but I guess I'm too shy to use it in the daily circumstances and in my work life.

00:04:58.427 --> 00:05:20.970
I'm responsible for more of a technical part of the product, so I'm responsible for the online service, everything that happens in the online service for upgrade, availability, lifecycle of environments, things of that nature, for our server, a server runtime and for our development tools and partner experiences.

00:05:20.970 --> 00:05:26.127
So this side of the Business Central product is part of my responsibility.

00:05:26.127 --> 00:05:29.327
Yannick is my counterpart.

00:05:29.327 --> 00:05:36.708
I've seen a fantastic show with you with you, gentlemen and Yannick, so lots of great insights there.

00:05:36.708 --> 00:05:46.466
So we work together, yannick on application and client experiences, and I'm more on the online service, server and partner tooling experiences.

00:05:47.548 --> 00:05:47.911
Excellent.

00:05:47.911 --> 00:05:56.906
It must be exciting to work with the online or the architecture, the technical platform of Business Central Online.

00:05:56.906 --> 00:06:01.108
Business Central Online is taking off.

00:06:01.108 --> 00:06:02.524
The product has taken off.

00:06:02.524 --> 00:06:03.507
It's an amazing product.

00:06:03.507 --> 00:06:07.708
I've seen some great news and information released about.

00:06:07.708 --> 00:06:10.160
You know the number of you know implementations there are.

00:06:10.160 --> 00:06:14.612
I believe I thought we saw like there was 40,000 plus or something, which is just amazing.

00:06:16.362 --> 00:06:16.983
And I can't.

00:06:16.983 --> 00:06:18.987
You know, I'm happy for it.

00:06:18.987 --> 00:06:20.913
It's been my entire life and I'm happy for it.

00:06:20.913 --> 00:06:36.449
But that must present some challenges, or architectural challenges for you as well, because you have this online environment with a number of users and users using it through web application, and you must have some challenges with performance.

00:06:36.449 --> 00:06:53.990
I know that a lot of individuals have security considerations or questions about having their information in the cloud, and then also some information or questions that I hear often are also even about the upgrades the upgrades between the architectural, the technical upgrade.

00:06:53.990 --> 00:06:55.826
I don't even know how to call it these days.

00:06:55.826 --> 00:06:58.209
I guess we used to call it the platform, but I guess now it's in the cloud.

00:06:58.209 --> 00:07:09.029
I'm not really certain what you would call it, but these are quite a bit of questions that we receive and we hear, so I was hoping to have the opportunity to talk with you about them.

00:07:09.439 --> 00:07:11.809
Excellent, excellent, very, very good topics, for sure.

00:07:11.809 --> 00:07:16.110
Yes, running service at that scale is an interesting experience for sure.

00:07:16.110 --> 00:07:23.730
For sure, and you just named all the core topics that we get to deal with quite often in our daily life.

00:07:24.632 --> 00:07:31.567
It must be both a challenge as well as excitement as well for you to be able to see that it works.

00:07:31.567 --> 00:07:38.353
We'll start with kind of a small portion of with Business Central Online.

00:07:38.353 --> 00:07:45.740
Now, one of the benefits of having an online product is that you have the upgrades and the updates as they become available.

00:07:45.740 --> 00:07:59.452
And with Business Central, the application, microsoft has a schedule of two major wave releases as of this recording two major wave releases per year and then monthly updates of the application.

00:07:59.452 --> 00:08:04.564
With the platform, is there a set schedule for the platform updates?

00:08:04.564 --> 00:08:07.266
With the platform, is there a set schedule for the platform updates?

00:08:07.266 --> 00:08:11.230
Do they coincide with the application or is there a different schedule that is followed for that?

00:08:12.151 --> 00:08:15.875
Yeah, yeah, so the way it works and you're completely right.

00:08:15.875 --> 00:08:23.367
So we have specific cadence with application updates and application updates apply to specific environments.

00:08:23.367 --> 00:08:32.448
So each customer have their own environment, which is a separate database of some sort that you can imagine, and application updates applied to each of these environments.

00:08:32.448 --> 00:08:45.192
And because these updates are applied to each of these environments, we provide this functionality for the environment owner to schedule these updates so they can decide when they want to apply the application update.

00:08:45.192 --> 00:08:59.659
Now you can imagine that on the technical level, on the server level, this infrastructure, the server itself which manages all of these environments, is shared across multiple environments.

00:08:59.659 --> 00:09:01.086
It's a multi-tenant system.

00:09:01.086 --> 00:09:07.501
So it's one NST managing multiple, one server tier managing multiple customer environments.

00:09:07.501 --> 00:09:20.042
So applying hotfixes and changes to the server cannot be scheduled by each individual environment owner because I mean that's a shared infrastructure provides resources for all of these shared resources.

00:09:20.042 --> 00:09:24.027
So we use different approach to that.

00:09:24.027 --> 00:09:30.096
So application platform hotfixes are rolled out as they come.

00:09:30.096 --> 00:09:46.851
So when developers produce certain changes, there is a process where they go through certain testing in different tiers of our infrastructure, so they obviously are tested by the developers themselves in their local environment.

00:09:46.851 --> 00:09:53.393
Then we have a so-called pre-production environment where the changes can be tested as well.

00:09:53.393 --> 00:09:59.793
Then they're merged into our official production branch and then they're scheduled to be deployed into production.

00:09:59.793 --> 00:10:11.615
So these changes also go through a thing which we call safe deployment, which is a very, very important practice for all of the cloud services.

00:10:12.179 --> 00:10:15.389
The hot fixes don't go to all of the customers at the same time.

00:10:15.389 --> 00:10:18.609
They go through a set of stages.

00:10:18.609 --> 00:10:29.110
So first they get deployed to a small segment of our service, to a few customers out there in a specific region, and then we monitor the health of these environments.

00:10:29.110 --> 00:10:38.028
If anything suspicious happens in that system, we interrupt the update, we analyze the changes and we introduce new things.

00:10:38.028 --> 00:10:51.408
But if we see that everything is healthy and is proceeding as usual, then we expand the deployment to more regions and more clusters and nodes in our service, eventually deploying all of it into all of the regions.

00:10:51.408 --> 00:10:54.804
So that happens because it's a shared infrastructure.

00:10:54.804 --> 00:11:03.908
That happens typically during the night, so we will locate night hours in each of the zones where we operate.

00:11:03.908 --> 00:11:09.275
And it is a challenge because we now run in some 170 countries.

00:11:09.275 --> 00:11:17.711
Yet in each Azure region there's obviously a good time zone where very little business activity.

00:11:17.711 --> 00:11:22.071
So we apply these fault fixes during that time period.

00:11:23.280 --> 00:11:38.506
Another dimension of kind of protection which we apply to this process is that our service nodes, where hosting our customers, they obviously don't consist of a single, you know, of a single node.

00:11:38.506 --> 00:11:40.307
They consist of multiple nodes.

00:11:40.307 --> 00:11:47.321
So when we apply our hotfixes, platform hotfixes, we apply them to a specific node at the time.

00:11:47.321 --> 00:12:01.014
So we start applying the hotfixes to one NST on one node and then this NST is taken out of rotation and all of the new connections are sent to the new environments and new NSTs.

00:12:01.014 --> 00:12:06.811
So the new customers logging in into BSC they go to that part of the service.

00:12:06.811 --> 00:12:18.234
We patch and update this NST and once it is healthy and we approve and the signals are coming in green, we open it up for our load balancer to send customers there as well.

00:12:18.234 --> 00:12:23.125
So this is happening face-by-face across these NSTs.

00:12:24.005 --> 00:12:32.850
However, I also have to admit that sometimes on that specific NST there are some users, so users doing jobs even during the night.

00:12:32.850 --> 00:12:38.186
They don't get any vacations, they don't get any night times or automated processes.

00:12:38.186 --> 00:12:42.350
There's some bad jobs, web services and portals doing the job.

00:12:42.350 --> 00:12:47.107
So we give a certain period of time to kind of drain these connections.

00:12:47.107 --> 00:12:52.509
There's a notification sent to the users and to the processes that we're going to interrupt you.

00:12:52.509 --> 00:12:54.267
Please refresh your browser.

00:12:54.267 --> 00:12:58.390
And that is a signal for us applying these hot fixes.

00:12:58.390 --> 00:13:01.769
So we try to avoid the disruption as much as we can.

00:13:01.769 --> 00:13:07.087
We try to thread as safely as possible across all of these instances.

00:13:07.087 --> 00:13:09.768
But, yeah, sometimes you can get this notification.

00:13:09.768 --> 00:13:16.908
Please refresh your browser, please reconnect, and that's the most disturbance that we apply to these environments.

00:13:17.399 --> 00:13:19.089
Yeah, that's basically the screen.

00:13:19.089 --> 00:13:26.909
That is a lot, and I've seen that message before too, and now it's nice to know and have an understanding of exactly what that means.

00:13:26.909 --> 00:13:30.610
You answered so many of the questions that I had in that explanation.

00:13:30.669 --> 00:13:31.272
Thank, you for that.

00:13:31.480 --> 00:13:47.206
Because having an online application that's available 24 hours a day, seven days a week, and if you have a shared update, as you had mentioned, across time zones, I know how difficult it is trying to coordinate with individuals over time zones.

00:13:47.206 --> 00:13:58.432
With you having users, it must be a challenge, but now I see that you isolate it by region to have their nighttime to minimize the disruption, which is good.

00:13:59.981 --> 00:14:06.668
And so sometimes users can see the same message popping up oh, you need to refresh your browser.

00:14:06.668 --> 00:14:15.052
For different reasons, like it's not necessarily that when you see this message is the time when we deploy an update.

00:14:15.052 --> 00:14:27.466
It could be done for other resource governance reasons, like if that specific machine or that specific segment of our service is experiencing more than usual traffic, we try to distribute it to other machines.

00:14:27.466 --> 00:14:41.187
And that's where we start to gradually kind of close it down and inform users please refresh your session and you will be sent to another machine with plenty of resources, lots of things you can do on another place.

00:14:41.759 --> 00:14:49.865
But yes, it may come in as a destruction, coming in when it comes during the day, for example during morning hours.

00:14:49.865 --> 00:14:57.807
But during morning hours and the daytime this should not be an update.

00:14:57.807 --> 00:15:12.625
So this is something else, this is resource governance and in fact we built a special chart which we monitor very tightly the number of times this notification is showed to the users.

00:15:12.625 --> 00:15:34.220
So in the live site, periodically we analyze this chart to see if any particular segment of our service is affected by this annoying dialogues more than usual, and then we try to understand what's going on and try to kind of check whether the resource situation is okay there or whether the updates are applying okay.

00:15:34.220 --> 00:15:37.308
So we monitor this as well, very, very tightly.

00:15:37.308 --> 00:15:42.048
But yes, sometimes it's unavoidable, so we'll have to gently push.

00:15:42.921 --> 00:15:48.493
When I see this pop up now moving forward, I'll start thinking about you and your team.

00:15:48.493 --> 00:15:50.187
Now it's like, oh, they must be doing something.

00:15:50.759 --> 00:16:09.711
Yeah, they see your blip on the chart, Like that's Christopher refreshing his session and also depending upon the time of the day, which also leads to another interesting point that you were talking about, where you may receive that notification if there's a lot of activity on a server or a service tier.

00:16:10.413 --> 00:16:14.869
For the sake of this conversation, I'll refer to it as the service tier performance.

00:16:14.869 --> 00:16:29.325
All right, so now that I mentioned you had you had mentioned that you may get this notification because there's a lot of activity on a particular service tier and you need to balance the load to ensure optimal performance for the users of the application.

00:16:29.325 --> 00:16:47.533
When talking with potential customers, prospects or customers who are looking to move from an earlier version of Dynamics Nav or even from an on-premise version of Business Central, a lot of times the topic of performance comes into play.

00:16:47.533 --> 00:16:53.089
Performance is always going to be, you know, a challenge or a topic, because there are many facets to performance.

00:16:53.089 --> 00:16:55.227
So let's talk about all things being equal.

00:16:55.227 --> 00:17:01.908
Take out any potential issues with coding or from the application side, right, because those sometimes could cause performance issues.

00:17:03.523 --> 00:17:15.130
From the platform side, or from the server side, the NST side, what considerations are in place for performance to allow for optimal use for a particular customer or user?

00:17:15.130 --> 00:17:19.310
And also, in the conversation we talked about how there is a shared environment.

00:17:19.310 --> 00:17:23.593
So with the shared environment, we can have many companies accessing that at the time of day.

00:17:23.593 --> 00:17:42.761
Um, so with the shared environment, we can have many companies, uh, accessing that at the time of day I may be a distributor and I need to have a high volume of transactions towards the end of the day because I need to fulfill my shipments, to get the trucks loaded in my packages out, and another retailer during a holiday season may have influxes during you know, uh, a time period.

00:17:42.761 --> 00:17:45.869
Uh, that's, um, you know, through a seasonal time period.

00:17:45.869 --> 00:17:58.512
Excuse me, what considerations or how do you address the performance when you have large loads coming at the NST in these shared areas?

00:17:59.173 --> 00:18:11.028
Yeah, yeah, yeah, very excellent question and yeah, we get to discuss this a lot and there's always a balance between performance and throughput, so these two topics come very closely together.

00:18:11.028 --> 00:18:16.020
What's the performance of Business Central at scale and how much throughput we can push through?

00:18:16.020 --> 00:18:17.949
Business Central at scale?

00:18:17.949 --> 00:18:21.348
And performance is an interesting beast.

00:18:21.348 --> 00:18:32.568
Very early on, when we just started building an online service, we understood that performance is going to be a mutual responsibility.

00:18:33.279 --> 00:18:48.035
I mean, I've also seen many fantastic webcasts from you gentlemen about performance and how much it is important to keep your code in a good shape and to use all of the tools which we provide to monitor performance.

00:18:48.035 --> 00:19:01.971
We provide many telemetry signals to understand where the bottlenecks may be in the code, in web service calls, in AL code, maybe there's some locking happening too much in specific tables.

00:19:01.971 --> 00:19:13.044
So we try to build a lot of tools for partners to understand where on which side of the performance stack, the problem may occur.

00:19:13.044 --> 00:19:36.652
So now you would like us to focus on kind of the resource governance part of this, because all of the other aspects were mentioned, but I would also want to sort of lean in just a little bit there, remind everyone that we did provide a lot of tools, fantastic tools to understand what is causing these issues.

00:19:36.652 --> 00:19:49.279
So there's an inclined performance profile at which you can run and see where the time is spent in certain process so you can understand whether a certain extension or certain piece of code or update changed the behavior.

00:19:49.279 --> 00:20:03.153
So I do want to kind of stress that performance is a multifaceted area and we take care of the resource governance part of it.

00:20:03.153 --> 00:20:15.685
Our goal and sole purpose of existence, work, life maybe there are a few other purposes, but our goal is to provide enough resources for all of these workloads.

00:20:15.685 --> 00:20:30.085
So, yes, so, while partner's responsibility and customer's responsibility to kind of keep the code sane, what we do, we designed the system in a way that it allows for scalability at many different layers.

00:20:30.085 --> 00:20:42.380
So the scalability bottlenecks may come, which has kind of presented itself in performance, may come from different places.

00:20:42.380 --> 00:20:44.327
So there's a scalability of database.

00:20:44.327 --> 00:20:53.030
So the database simply cannot cope with that many requests coming, for example, from a web store, from e-commerce solution.

00:20:53.030 --> 00:21:02.029
So there is a scalability of compute and scalability like the traffic which is going between these systems.

00:21:02.029 --> 00:21:07.550
So there's many aspects and on each of this level we build automatic scaling system.

00:21:09.521 --> 00:21:15.114
So for database scalability we use a system of tiers.

00:21:15.114 --> 00:21:32.036
So there are several tiers in our infrastructure which we use to kind of move the database between these tiers as it exhausts its limits in the specific tier.

00:21:32.036 --> 00:21:35.690
So what happens is that this system is fully automated.

00:21:35.690 --> 00:21:40.271
There's no way any human being is capable of monitoring this in real time.

00:21:40.271 --> 00:21:47.240
There is automation which is monitoring each and every environment time.

00:21:47.240 --> 00:21:52.921
There is automation which is monitoring each and every environment and it tries to understand whether this environment is getting close to the limits of what this database can handle.

00:21:52.921 --> 00:21:58.733
So not when it crosses these limits, but when it's getting close to these limits.

00:21:58.733 --> 00:22:05.713
So when we detect this moment of time, we take this database and move it automatically into another tier.

00:22:05.713 --> 00:22:12.391
So it gets moved into a quieter place, into a place with more resources.

00:22:12.391 --> 00:22:14.054
It scales as well.

00:22:14.054 --> 00:22:16.224
Azure SQL has many, many different tiers.

00:22:16.224 --> 00:22:27.119
Azure Elastic Pools, which we also use on the back end, also have a lot of configuration settings to define how much capacity you want to allocate to each environment.

00:22:27.119 --> 00:22:38.476
So your database goes into another tier automatically and by that you get more throughput and thereby more performance for your processes.

00:22:39.881 --> 00:22:43.210
Now, so that is the database situation.

00:22:43.210 --> 00:22:47.009
The same happens for the compute capacity.

00:22:47.009 --> 00:22:55.592
You said correctly that the compute is shared, so many environments using compute on our service nodes.

00:22:55.592 --> 00:23:18.194
So when we see that the traffic is approaching some limits of CPU and other capabilities of this node which are about to be uncomfortable, we start scaling out the service to provide more of these resources and we stop sending sessions to this segment of the service.

00:23:18.194 --> 00:23:23.268
So all of the new connections go to the new services, the new services.

00:23:23.288 --> 00:23:46.775
So whenever you experience this slowness and something which is maybe caused by the resources like in the worst case, refreshing your browser, trying to kind of hard refresh your browser, close, reopen, so that should automatically redirect your session to another machine compute-wise.

00:23:46.775 --> 00:23:49.528
So that's another good advice.

00:23:49.528 --> 00:23:51.887
But all of it is happening automatically.

00:23:51.887 --> 00:23:58.983
We kind of redirect resources to machines with enough capacity and to database and pools with enough capacity.

00:23:58.983 --> 00:24:22.888
And also I wanted to say that, looking at our KPIs, so we try to measure how long time each database spends on, or each user or each session spends on the machine with insufficient capacity, how long time users are suffering, and that percentage of time is very, very small.

00:24:22.888 --> 00:24:31.560
So we are in 99.9x of sending sessions to the machines with enough capacity.

00:24:32.405 --> 00:24:37.417
And what I'm trying to say by that is that when the theme comes to the performance and the throughput.

00:24:37.417 --> 00:24:44.233
It's rarely caused by insufficient CPU memory and database throughput.

00:24:44.233 --> 00:24:47.799
It's quite often in some other segments.

00:24:47.799 --> 00:24:51.365
Insufficient CPU memory and database throughput, it's quite often in some other segments.

00:24:51.365 --> 00:24:59.548
So the combination of database locking, extensions too many instructions issued at the same time.

00:24:59.548 --> 00:25:01.930
Sometimes it is throttling as well.

00:25:01.930 --> 00:25:12.976
So you kind of send so many signals to Business Central that at some point we start throttling and capping this kind of influx of requests.

00:25:13.536 --> 00:25:16.397
That was the question that I had, if there was a ceiling.

00:25:16.397 --> 00:25:30.346
We talked about the scaling of performance and being able to work with processing and CPU and then also with the ability to move the database, and I do.

00:25:30.346 --> 00:25:31.209
I'll reiterate what you said is before.

00:25:31.209 --> 00:25:46.597
That's what I was trying to say at the beginning is is there are application considerations to performance and I can say from a lot of times I get a lot of performance issues and sometimes it could be, as you had mentioned, multiple extensions that are installed that may be competing with each other, or even if somebody had created some code that may not be the most efficient.

00:25:46.597 --> 00:25:55.796
You know, even if I put something in there and tell it to wait a minute before it does the next process, a user may feel that it's slow performing, when in reality it was.

00:25:55.796 --> 00:25:56.597
It was written that way.

00:25:57.266 --> 00:26:06.367
And you mentioned the performance, the incline performance analyzer, which I think I did see in 2024, wave two, is going to also have some enhancements.

00:26:06.367 --> 00:26:09.076
I read the notes last week because I'm excited about that.

00:26:09.076 --> 00:26:19.391
I do use that regularly as well because it is a good tool to help pinpoint from the application side the amount of time that's spent in certain areas, to help you isolate your code.

00:26:19.391 --> 00:26:22.077
So I did, taking the code piece into consideration.

00:26:22.077 --> 00:26:23.280
That's where I was wondering.

00:26:23.280 --> 00:26:45.486
Taking the code piece into consideration, that's where I was wondering if there was a cap, because if someone does create an extension that could impact performance, as you had mentioned, maybe just send, you know, an over abundant amount of requests, where is that cap that it would actually now say, okay, well, you can't you know you don't have an infinite pool of resources, right?

00:26:45.566 --> 00:26:52.694
I mean theoretically, within Azure and an online environment, it's supposed to scale up to give you the resources adequate for you to perform your functions.

00:26:52.694 --> 00:27:03.353
But if there is some incorrectness or something going on, that's, you know, inundating or creating excessive traffic, how do you determine the ceiling of that?

00:27:04.074 --> 00:27:04.315
Yeah.

00:27:04.315 --> 00:27:15.737
So very, very good question and I think, like I mentioned, our goal and aspiration and the whole team is kind of rallying behind that is to provide enough resources to everyone.

00:27:15.737 --> 00:27:18.006
I mean, this is the sole purpose of the online service.

00:27:18.006 --> 00:27:20.352
You cannot starve on resources.

00:27:20.352 --> 00:27:27.769
So a lot of efforts were put into place to kind of automate this process and make sure that it doesn't happen.

00:27:27.769 --> 00:27:34.299
But for some scenarios we wanted to introduce a reasonable ceiling.

00:27:34.299 --> 00:27:57.499
So, for example, for web service requests due to just heavy workloads, like extremely heavy workloads or a programming mistake or inefficient code in this web service request, you may start consuming unusually high number of resources in Business Central.

00:27:57.499 --> 00:28:07.657
So what we apply here is a throttling limit which I believe is 6,000 requests per five-minute sliding window.

00:28:07.657 --> 00:28:14.617
So five-minute sliding window if you issue more than 6,000 requests, we start throttling them.

00:28:14.617 --> 00:28:16.491
So that's for web service requests.

00:28:17.244 --> 00:28:23.939
And the way we arrived to that number is we kind of went through the same safe rollout process.

00:28:23.939 --> 00:28:26.704
We kind of went through the same safe rollout process.

00:28:26.704 --> 00:28:31.977
So we introduced the measurements in our system to understand how much throughput different workloads require.

00:28:31.977 --> 00:28:38.092
So we monitored the system for a very, very long period of time I would say even more than a year.

00:28:38.092 --> 00:28:56.525
So more than a year we were looking at how much resources and how many requests customers are sending to us and then we collected all of this telemetry and then we looked at the peaks, like what is the throughput that they actually use, and then we almost doubled it generally.

00:28:57.446 --> 00:29:18.730
So we try to be super generous in this throttling threshold to make sure that in this throttling threshold, to make sure that going to these limits it really is something where it's either by mistake or something unusually high load coming from some e-commerce solution or some reporting solution.

00:29:18.730 --> 00:29:33.993
So that's how these limits were defined by us looking at telemetry, trying to see even the heaviest workloads and putting these thresholds even above these heaviest workloads so they should be sufficient for the vast majority of our customers.

00:29:33.993 --> 00:29:44.914
Another unit of measure which we're tracking is the number of scheduled tasks or job queues, so you can create some processes looping automatically and doing things.

00:29:44.914 --> 00:29:59.712
We also apply a limit of five scheduled tasks per user executed in parallel, so you cannot spawn too many tasks or execute too many per environment at the same time.

00:29:59.712 --> 00:30:05.618
So with these numbers, hopefully there's enough throughput for many, many, many workloads.

00:30:05.618 --> 00:30:06.484
Yeah.

00:30:08.467 --> 00:30:19.085
Just to touch base on the workloads and the limits and one key point that you're mentioning that is a common practice for all online environments.

00:30:19.085 --> 00:30:25.328
I have had interfaced with other applications from Business Central, interfaced with other applications from Business Central.

00:30:25.328 --> 00:30:25.892
You mentioned web requests.

00:30:25.892 --> 00:30:31.348
Well, we had web requests where we had to, again just by the nature of business, restructure how we were sending information because they also had limits.

00:30:31.348 --> 00:30:39.455
So the limits are there and not to say that there's a limit to the application, it's to ensure that you have proper performance and throughput.

00:30:39.455 --> 00:30:48.440
And, as you had mentioned, in the case where we had ran into the problem, we found that they had a measure in place where, if you sent the transactions differently, you wouldn't hit that throughput.

00:30:48.440 --> 00:30:50.854
We were sending them individually instead of in batch.

00:30:51.365 --> 00:30:56.498
It goes to the point where some of these I just want to just put people in mind we talk about limits and their ceilings.

00:30:56.498 --> 00:30:57.809
It's not a negative to me.

00:30:57.809 --> 00:31:02.596
It's a positive because it can help ensure that everybody has the equal processing to continue.

00:31:02.596 --> 00:31:18.756
It can help ensure that everybody has the equal processing to continue, but also it may be a signal that there may be something else that you should maybe look at if you're constantly hitting those limits and that's what we had found in this transaction is the web service application interface that we were tasked to take a look at.

00:31:18.756 --> 00:31:26.448
Somebody was just sending the same transactions multiple times individually, instead of in one batch, with like they thought they were.

00:31:26.448 --> 00:31:30.683
So the the throttling, as we'll call it, or the limits that they set.

00:31:30.683 --> 00:31:44.213
The rate limits, as you mentioned, usually are adequate for normal processing, and if you hit those limits a lot, maybe it's time to maybe take a look at the process to see if you're using it effectively will they get notified if they're hitting that threshold too many times?

00:31:47.644 --> 00:31:48.654
Yes, yeah, they receive a response from the service.

00:31:48.654 --> 00:31:51.752
429, if I'm not mistaken, like wait a little bit.

00:31:51.752 --> 00:31:54.388
Wait, you've sent in too many requests.

00:31:54.388 --> 00:31:57.594
So yeah, and that's absolutely a good point.

00:31:57.594 --> 00:31:59.833
Brett, this is the time to re-evaluate.

00:31:59.833 --> 00:32:04.326
Are you doing something not right or are you just?

00:32:04.326 --> 00:32:17.561
You can spread your workload over a longer period of time, move it to another, to offline mode, to kind of async mode, or batch different requests or make requests shorter.

00:32:17.561 --> 00:32:19.708
There's many ways of coping with this.

00:32:19.708 --> 00:32:37.805
There's many ways of coping with this, but, to your point, this is more of a limit to ensure fairness rather than not provide enough resources, because we absolutely do want to provide enough resources for all SMBs.

00:32:37.805 --> 00:32:44.798
And, by the way, when we talk SMBs, you would be surprised what kind of SMBs we run in Business Central Service.

00:32:44.798 --> 00:32:49.556
Yes, some years ago we started with smaller companies.

00:32:49.556 --> 00:32:55.798
You know 20, 15, 20, 30 users, companies joining the cloud.

00:32:55.798 --> 00:32:57.289
So that was years ago.

00:32:57.289 --> 00:33:02.173
These days, smbs come in really, really large.

00:33:02.173 --> 00:33:04.479
You know form and shape.

00:33:06.606 --> 00:33:08.835
I saw an interesting statistic.

00:33:08.835 --> 00:33:13.755
I think it was at Directions, not last year but the year before.

00:33:13.755 --> 00:33:15.199
There was a presentation.

00:33:15.199 --> 00:33:17.553
I'm getting old so I forget a lot of things.

00:33:17.553 --> 00:33:28.693
Someone put a presentation for large implementations or you know, large business central implementations, and I think they had a statistic, and this is me saying from memory for what I thought I saw.

00:33:28.693 --> 00:33:31.648
If you can validate it, great, if you can't, great.

00:33:31.648 --> 00:33:40.340
I think they had mentioned, like the largest implementation at 8,000 users or some significant number, which I was quite impressed.

00:33:40.340 --> 00:33:46.213
You know, sometimes people think of the SMB market they think of, well, the application can only handle the 20 to 30.

00:33:46.213 --> 00:33:54.696
I mean, I know firsthand of somebody using it with, you know, 400 users, which is a nice size implementation it is.

00:33:54.696 --> 00:34:00.325
So it's not the small, but I was impressed to see that there was thousands of users, if I recall the statistic.

00:34:00.325 --> 00:34:04.372
And if I'm incorrect then you know, we'll just chalk it up to me being old.

00:34:04.432 --> 00:34:13.990
Yeah, the spectrum is very wide and you're absolutely right 300 users, 400 users is absolutely not unusual.

00:34:13.990 --> 00:34:18.498
This is almost the usual situation in Business Central.

00:34:18.498 --> 00:34:33.047
So the question is what do the users do, how they interact with the service, how they work with it, like, what are the web service workloads, what are the end users workloads, what are the integrations doing the reporting doing?

00:34:33.047 --> 00:34:34.893
That's where most of the discussion is.

00:34:34.893 --> 00:34:39.431
The number of users is not really a limiting factor in Business Central.

00:34:39.851 --> 00:34:41.536
You're absolutely correct.

00:34:41.536 --> 00:34:44.630
I've always had the question people say how many sales orders can you handle?

00:34:44.630 --> 00:34:50.027
And I said it's not the number of sales orders, it's, in essence, the number of lines you want to process.

00:34:50.027 --> 00:34:57.369
If you're processing one sales order with 20,000 lines, or if you're processing 20,000 sales orders with one line, it's different, but it's not the sales order.

00:34:57.369 --> 00:34:58.365
It's like you had mentioned.

00:34:58.365 --> 00:34:59.706
It's the workload that you're working on.

00:35:00.206 --> 00:35:11.833
By the way, since we are on this topic, I also wanted to advertise one of the resources that we have online which I would definitely recommend everyone to scan and to read about the throughput and performance.

00:35:11.833 --> 00:35:21.719
We use this short URL, akams, and the URL is called akamsvcservice slash VC service.

00:35:21.719 --> 00:35:37.134
So if you navigate to that resource, we have three segments there One talking about the high availability and disaster recovery and business continuity, so how we make sure that all of your data remains accessible and available at all times.

00:35:37.134 --> 00:35:41.110
The second one is scalability, the things which we're talking about right now.

00:35:41.110 --> 00:35:42.275
What is the throughput?

00:35:42.275 --> 00:35:50.458
How we scale the resources With practical examples how much customers pushing through Business Central today.

00:35:51.425 --> 00:36:08.478
So we collected some telemetry how many sales orders to your point with how many lines people post and create through Business Central, and we put these telemetry results on that page so you can see the examples from the live customer installations.

00:36:08.478 --> 00:36:09.358
They're all anonymized.

00:36:09.358 --> 00:36:15.934
We will never share any private information of our customer, but you can see examples from the real-life telemetry.

00:36:15.934 --> 00:36:27.952
And the third one is our live site, our operational processes, how we deal with incidents, with support and other aspects of running the live service.

00:36:27.952 --> 00:36:36.454
So that is a very, very good resource BC Service very easy to find akms forward slash BC Service.

00:36:36.644 --> 00:36:38.753
I will definitely put that up and keep it up.

00:36:39.206 --> 00:36:39.427
Is it a?

00:36:39.467 --> 00:36:40.913
real-time update on the statistics.

00:36:40.913 --> 00:36:42.391
So should I keep it open?

00:36:43.126 --> 00:36:44.952
Yeah, we need to do better job.

00:36:44.952 --> 00:36:52.036
But this, the last one, is fairly recent, so it it was put there within the time frame of directions.

00:36:52.036 --> 00:36:58.898
So, um, yeah, but as the service evolves, you can imagine that these numbers are already too weak.

00:36:58.898 --> 00:37:04.623
So it only goes in the, in the in the other direction, in the other direction, in the growing direction.

00:37:04.623 --> 00:37:07.954
No, it's good If you see the numbers there and they fit your business.

00:37:07.954 --> 00:37:13.016
So that's a safe choice to further assess Business Central.

00:37:13.016 --> 00:37:15.012
Try it out and do the implementation.

00:37:16.467 --> 00:37:23.025
No, those are good statistics for somebody to look at to give a general idea if they have a good understanding of the business.

00:37:23.025 --> 00:37:27.416
But it should not be the only consideration they use if they want to use the product.

00:37:27.416 --> 00:37:34.545
It's just one signal that they can look at, because it's important to make sure that they're looking at the right information to make the right decision.

00:37:34.545 --> 00:37:40.117
So talking with somebody about it would make it a proper decision.

00:37:41.125 --> 00:37:42.226
Sorry.

00:37:42.226 --> 00:37:48.518
I just wanted to say that the way we built this article is we have this program called Concierge.

00:37:48.518 --> 00:37:58.099
So this program is meant to provide some consultancy and help to larger customers do cloud implementation.

00:37:58.099 --> 00:38:15.391
So if you are a larger customer with more than I believe it's 50,000 revenue or something like that yearly revenue licenses so you can ask for concierge service to provide you with guidance what business central is, what it can handle, what's the scalability considerations.

00:38:15.391 --> 00:38:23.181
So for a long period of time that team was assembling all of the questions and you know from these larger companies.

00:38:23.181 --> 00:38:26.231
So we took all of this feedback and packaged into this article.

00:38:26.231 --> 00:38:32.032
So now everyone has access to the same answers that we were giving to our larger prospects.

00:38:32.385 --> 00:38:33.891
So you get concierge service.

00:38:33.891 --> 00:38:34.432
That's awesome.

00:38:35.164 --> 00:38:40.418
That is a great resource, then, to have all that information on the akams4.bc service.

00:38:40.418 --> 00:38:42.353
We have to look at that and use that, chris.

00:38:42.393 --> 00:38:48.092
Chris, that's a good one to show prospects and customer potential customers, even customers looking to upgrade.

00:38:48.092 --> 00:38:49.476
I mean, there is uh.

00:38:49.476 --> 00:38:52.793
That's that that's a big portion of uh, the BC implementation as well.

00:38:52.793 --> 00:39:06.016
It's not only new prospects, but also customers looking to migrate to business central online uh with online services and um multi-tenancy shared environments.

00:39:06.016 --> 00:39:21.172
Uh, one of the questions that I mentioned when we have conversations with many individuals and questions that I have even more, so you start hearing about issues with technology on a wide scale is security.

00:39:21.172 --> 00:39:37.251
So then let's talk about, from the NST point of view, how can we ensure that individuals' information, individual businesses' data as you had mentioned, you can anonymize some statistics.

00:39:37.251 --> 00:40:00.978
But also, now that we have shared environments, we have users connecting online, what assurances do they have that this is a secure system, yeah, that their data is secure, that the information is secure, that if, um, you know, somebody can't compromise the system from the cloud-based uh avenue, um this is.

00:40:01.018 --> 00:40:21.275
This is an excellent topic, I think, like these days, cyber cybersecurity is so high on every agenda With all the troubles happening in the world and all of the cyber attacks and businesses getting in trouble with cyber attacks.

00:40:21.275 --> 00:40:23.958
This is an important point to keep in mind.

00:40:23.958 --> 00:40:25.918
Cyber attacks this is an important point to keep in mind.

00:40:25.918 --> 00:40:27.780
So there are two ways of answering this.

00:40:27.780 --> 00:40:32.922
So there's an official way which is fairly standard.

00:40:32.922 --> 00:40:39.128
So Business Central undergoes certifications recurrently.

00:40:39.128 --> 00:40:46.784
So every year there's a certification process assessing the way Business Central manages the service and manages the customer data.

00:40:46.784 --> 00:40:57.992
So there is ISO certifications which Business Central goes through with different authorities and provides proof of how these processes are executed.

00:40:57.992 --> 00:41:11.791
So you can see some ISO 27001, see some ISO 27001, some standards mentioned on our akamsbc security page.

00:41:11.791 --> 00:41:13.315
So all of these certifications SOC compliance, iso compliance.

00:41:13.315 --> 00:41:29.025
So these certifications are listed on that page and in the scope of these certifications, there are questions about whether you use encryption, whether you use encryption of databases at rest, whether you encrypt everything that moves through the traffic.

00:41:29.025 --> 00:41:43.759
So all of the industry standard practices for security are not all, but many are kind of part of this ISO certification criteria, so NSOC compliance criteria.

00:41:43.759 --> 00:41:47.829
So that's a good resource for prospects and customers to assess.

00:41:49.166 --> 00:41:55.360
Another part of this answer is quite interesting.

00:41:55.360 --> 00:42:04.373
So I don't know if you get this feeling, but it feels like Business Central Team produces a lot of new capabilities.

00:42:04.373 --> 00:42:13.936
You know, every release I don't know if you get this feeling Every time I go to directions and I see the keynote and concurrent sessions every half a year.

00:42:13.936 --> 00:42:15.338
Actually there's just so much.

00:42:15.338 --> 00:42:34.652
There's new capabilities, there's new features, there's new, you know, like online service capabilities as well coming out, and yet a lot of effort that is going on here internally is actually put on the work done in the security area.

00:42:34.652 --> 00:42:48.135
So I'm not going to call a percentage of the team resources working on security topics, but it's a huge percentage of the team working within the realm of security.

00:42:48.135 --> 00:42:56.577
So ERP is a jewel in the business world, so that data needs to be protected.

00:42:56.577 --> 00:43:05.510
Data needs to be protected, and the amount of work that the team here and across Microsoft teams puts into security is incredible.

00:43:17.735 --> 00:43:20.869
And we are in a very good situation because in a large organization like ours, there's this principle that when one is attacked, everyone is attacked.

00:43:20.869 --> 00:43:34.224
So whenever a product from Microsoft portfolio gets exploited or even potentially exploited in any way, so this information is propagated automatically to many teams, to all teams within Business Central.

00:43:34.224 --> 00:43:47.246
So there are tools and there are processes in place to share this information, to react on these incidents or suspected incidents and apply changes rapidly across all of the other services.

00:43:47.246 --> 00:43:51.081
So this is what we monitor and do pretty much all the time.

00:43:51.081 --> 00:44:10.766
So there's always new attack vectors emerge and new approaches and new ways to penetrate systems, and the team is very agile in monitoring this situation and applying these changes to the system, and so this is a lot of work of Business Central.

00:44:11.375 --> 00:44:23.302
And, yes, the way we design our service in the cloud, we make absolutely certain that this service is sealed from all of the external factors that we are aware of.

00:44:23.302 --> 00:44:26.197
First of all, there's no external connections.

00:44:26.197 --> 00:44:31.762
Everything is sealed within these nodes of the service.

00:44:31.762 --> 00:44:44.967
Only the services whitelisted and registered to work with resources have access to resources, also known as managed identity way to kind of authenticate to these resources.

00:44:44.967 --> 00:44:49.827
So there's a lot of work going on in this space, for sure.

00:44:49.827 --> 00:45:07.860
And of course, in multi-tenant system, each customer environment is a separate, database is a separate environment and the data is never in the place where one tenant can kind of somehow get access to the data of another environment.

00:45:07.860 --> 00:45:12.228
It's just by design sealed off into complete silos.

00:45:12.228 --> 00:45:16.101
The data is encapsulated in the database.

00:45:16.101 --> 00:45:28.389
Everything that manages that data is running in its own process in its own sealed kind of entity and is never shared across these environments.

00:45:28.389 --> 00:45:32.125
That is the main principle of designing multi-tenant services.

00:45:32.125 --> 00:45:32.686
Okay.

00:45:32.934 --> 00:45:47.806
So they have the same shared application, but the database and their tenant is isolated, so it's separate, correct, as you mentioned, it's sealed, so the shared metadata all of the properties, the pages, the code units, the code.

00:45:48.516 --> 00:45:53.463
So that is obviously shared, but when it comes to data, that is managed completely separately.

00:45:54.224 --> 00:45:58.284
Okay, yeah, that is an important point to note.

00:45:58.284 --> 00:46:05.186
And again, everyone gets a little nervous with security, even recently this week, because I got stuck, I couldn't fly anywhere.

00:46:05.186 --> 00:46:08.204
I wasn't security-related directly but indirectly.

00:46:08.204 --> 00:46:10.097
Yeah, well, interestingly enough.

00:46:10.137 --> 00:46:16.208
I would say that this was more of an availability incident than security.

00:46:16.668 --> 00:46:17.329
Yes, yes.

00:46:19.001 --> 00:46:33.467
But it kind of brings this point of awareness that that, uh yeah, we also need to remember about security as well as availability, high availability correct, correct and and sometimes, as you had mentioned, the lines get blurred between what it was, I mean we.

00:46:33.527 --> 00:46:42.862
I think by this point, everybody understands, or has at least read, what had happened, but it still raises the question of okay, well, what would happen to my environment if something like that were to happen?

00:46:42.862 --> 00:46:46.922
You talk about the databases.

00:46:46.922 --> 00:47:00.202
Another thing that we often hear and are asked is now this database, my information's online, it's in the cloud, in this great big sky place.

00:47:00.202 --> 00:47:02.257
What about?

00:47:02.257 --> 00:47:04.123
We talked about the data security.

00:47:04.123 --> 00:47:18.570
What about backups and data redundancy, and we talked about the data protection in a sense, but is it user-based for the backups or data redundancy, so that if something were?

00:47:18.610 --> 00:47:24.884
to happen in one particular Azure region to a center where maybe my data is stored?

00:47:24.884 --> 00:47:29.490
How do I ensure that I have access to that data if they were to have an incident at that facility?

00:47:29.490 --> 00:47:33.543
As well as now, just backups, if I have a problem with my system and I need to restore it on my own.

00:47:34.184 --> 00:47:35.447
Yeah, yeah.

00:47:35.447 --> 00:47:45.188
So that's again the topic of availability of data, and you know this is also one of the top priorities of the team, obviously.

00:47:45.188 --> 00:47:50.666
So how can we make this data available to everyone running DC?

00:47:50.666 --> 00:47:55.806
So that is also kind of a multifaceted space.

00:47:55.806 --> 00:48:19.684
I don't even know where to start, but the amount of replication that we apply and redundancy that we apply to data to make sure that it gets accessible and available even in case where the Azure region goes offline, which is happening very, very rarely is enormous.

00:48:20.344 --> 00:48:25.490
So there's many, many kind of levels to ensuring the data availability.

00:48:25.490 --> 00:48:30.586
First of all, we use, in the backend, we use the technology Azure SQL.

00:48:30.586 --> 00:48:35.646
So Azure SQL on its own provides a lot of redundancy on the physical level.

00:48:35.646 --> 00:48:48.931
So what we see is a logical database, but on a physical level, all of these files and the actual servers where the data is hosted and is run it's all replicated and kept up to date and kept highly available.

00:48:48.931 --> 00:48:59.409
Azure SQL Database on its own has 99.99, like four nines availability SLA.

00:48:59.409 --> 00:49:01.141
So Azure SQL Database has four nines availability SLA.

00:49:01.141 --> 00:49:03.034
So Azure SQL Database has four nines availability SLA.

00:49:03.034 --> 00:49:05.523
It's always highly available.

00:49:06.956 --> 00:49:16.681
Now, what we also apply to Business Central is a feature called a point-in-time restore, which is one of my personal favorites.

00:49:16.681 --> 00:49:28.489
So with this feature, which is available in Business Central Admin Center, you can go back in time restoring your database to any point in time within the last 28 days.

00:49:28.489 --> 00:49:42.288
So if anything happens to your data, let's say that by coincidence or by chance or not by chance, you installed an app or you ran a process and it somehow damaged your data.

00:49:42.288 --> 00:49:45.684
So something went wrong and you cannot simply undo that.

00:49:45.684 --> 00:49:54.483
So you can go to any minutes before you know in the last 28 days and recover your data to that point in time.

00:49:54.483 --> 00:49:56.594
This is a mind-blowing capability.

00:49:57.960 --> 00:50:01.902
That is mind-blowing and the first question.

00:50:01.922 --> 00:50:05.846
If you can pick a minute up, I'm sorry I just get excited about some of these topics, and it's it's.

00:50:05.846 --> 00:50:22.644
I get excited because these are real world situations and and instances that occur, because I can't tell you how many implementations that I've gone through on premise that they've had an incident and they've had to restore to a backup from the day before.

00:50:22.644 --> 00:50:26.945
They've had to restore from a backup from if they have a valid backup.

00:50:26.945 --> 00:50:31.320
I'm not even going to get into those that think they're doing backups and they realize they don't have a backup.

00:50:33.936 --> 00:50:34.597
Separate topic.

00:50:34.597 --> 00:50:42.507
You know to have a good disaster recovery plan, but to be able to go to the minute for 28 days is amazing.

00:50:43.036 --> 00:50:44.000
The simplicity too.

00:50:44.000 --> 00:50:48.965
It's going to the admin center and choosing a specific date and time.

00:50:48.965 --> 00:50:52.304
And imagine if you have to manage that on-prem.

00:50:52.934 --> 00:50:54.159
We'll pretend I'm a prospect.

00:50:54.159 --> 00:50:55.184
What does that cost?

00:50:55.184 --> 00:50:58.322
Is that an additional cost or is that included as part of the Business Central license to have that option?

00:50:58.322 --> 00:50:59.931
That is included, so, or is that included as part of the business central license?

00:50:59.951 --> 00:51:00.333
to have that option.

00:51:00.333 --> 00:51:00.675
That is included.

00:51:00.675 --> 00:51:02.400
So that is all included.

00:51:02.400 --> 00:51:06.411
There's no, that's the magic right there.

00:51:06.914 --> 00:51:09.503
Yeah, that's the magic right there that you can roll back.

00:51:09.684 --> 00:51:12.538
I mean, there's just so much which is in that package.

00:51:12.538 --> 00:51:13.701
Yes, yeah, for sure.

00:51:13.721 --> 00:51:17.538
Yeah, that's um so for 28 days.

00:51:17.617 --> 00:51:18.882
You can go to a point in time.

00:51:18.882 --> 00:51:21.067
I'm sorry you got me excited now with this.

00:51:21.067 --> 00:51:22.760
Go on, go on.

00:51:22.760 --> 00:51:31.128
Yes, what about could you go to a previous point in time before the 28 days?

00:51:31.128 --> 00:51:38.824
Like if there's a weekly backup, a monthly backup or any other disaster recovery plan if the customer needs it for whatever reason?

00:51:38.824 --> 00:51:47.027
I mean, I think a minute to a specific minute for the previous 20 days should satisfy most, yeah, needs for recovery.

00:51:47.027 --> 00:51:50.425
But if somebody needs to go further, are there any options available?

00:51:50.987 --> 00:51:51.208
yes.

00:51:51.208 --> 00:51:53.215
So another option which is available you can.

00:51:53.215 --> 00:51:56.460
You can download the entire database in the backpack format.

00:51:56.460 --> 00:52:07.737
So somewhat technical, but you can go and download the entire database into your storage and then store it forever for 10, five, 20 years.

00:52:07.737 --> 00:52:10.003
This is now your responsibility.

00:52:10.003 --> 00:52:18.922
Now, restoring data beyond 28 days is not a process which is happening a lot.

00:52:18.922 --> 00:52:40.903
So for these purposes, we recommend to use our cloud migration tool, which is a bit of a I would call it is a bit of a workaround, but again, it also happens in extreme circumstances where you need to recover environment which is more than 28 days old.

00:52:40.903 --> 00:52:43.978
So, yes, you can take that back up.

00:52:43.978 --> 00:53:01.168
You can create an on-prem environment or just install unravel this database to your local SQL server, connect to a new environment in the cloud, migrate the data, clean up the data and you can still recover your data even from beyond 28 days.

00:53:01.168 --> 00:53:07.226
So that is a possibility and hopefully not many customers will have to revert to that possibility.

00:53:08.416 --> 00:53:16.297
But the way we came up with 28 days is that we, as we usually do with many of our features, we surveyed many partners.

00:53:16.297 --> 00:53:27.001
You know how far back are you recovering or restoring your data, and hardly anyone went beyond one week, like if you go beyond one week, it has to be something really dramatic.

00:53:27.001 --> 00:53:31.856
So we said, okay, 28 days should probably be enough, so not one week.

00:53:31.876 --> 00:53:36.460
but we set it to 28 days, that's plenty, and that's also what.

00:53:36.762 --> 00:53:42.826
Azure SQL Server kind of provides us with, so that's a very, very good availability option.

00:53:43.076 --> 00:53:46.192
Any minute in time for four weeks is significant.

00:53:46.192 --> 00:53:52.766
Yeah, and I know many with disaster recovery plans, with rotations of tapes, with daily, with weekly, with monthly backups.

00:53:52.766 --> 00:53:55.461
Whatever plan they have, they're not going to get a point in time.

00:53:55.996 --> 00:53:57.563
They're going to get a specific day.

00:53:58.295 --> 00:53:59.137
In many cases.

00:53:59.257 --> 00:54:05.509
If you need more than a week, it's because that's the last full backup you had, Not because that was the only option.

00:54:05.509 --> 00:54:07.342
Yeah, that's the only option.

00:54:07.996 --> 00:54:08.539
Yeah, good point.

00:54:08.539 --> 00:54:09.001
Good point.

00:54:09.001 --> 00:54:11.280
That's not it, by the way.

00:54:11.280 --> 00:54:22.884
So, interestingly, this is a way for you to kind of recover your data if something happens within your environments, like something happened to your environment.

00:54:22.884 --> 00:54:30.023
But we also protected your customer data from Azure outages.

00:54:30.023 --> 00:54:39.442
So, like I said, they happen very rarely, but when they happen, we want to make sure the data still remains available, and it's a very interesting story.

00:54:40.538 --> 00:54:53.099
So some time ago, we had this incident where one of the Azure regions had troubles with I believe it was something with the cooling system.

00:54:53.099 --> 00:55:04.605
So unfortunately, the environments in that region were not available for a continuous time, so the Azure region was not available and therefore all of the environments in that region were not available.

00:55:04.605 --> 00:55:06.181
So that was quite some time ago.

00:55:06.181 --> 00:55:23.695
I have to do this full disclaimer, but instantly following this incident, just like we do for any severe incidents like this one, the team redirected the efforts to build support for a capability called Azure Availability Zones.

00:55:23.695 --> 00:55:39.800
So the way the Azure is kind of laying out their Azure regions is that every Azure region let's just say East US or East US 2, it's not one data center, it's a collection of data centers.

00:55:39.800 --> 00:55:41.262
It's not one data center, it's a collection of data centers.

00:55:41.262 --> 00:55:46.346
There's many data centers, buildings hosting these machines or keeping these machines I use.

00:55:46.425 --> 00:55:47.226
US2, by the way.

00:55:47.507 --> 00:55:49.869
There you go, so let's use that as an example.

00:55:50.108 --> 00:55:50.670
East east.

00:55:50.829 --> 00:56:00.922
Yeah, so that's a collection of data centers and not only it's a collection of data centers.

00:56:00.943 --> 00:56:04.398
It's a collection of availability zones, so groups of data centers, of which there is always three.

00:56:05.340 --> 00:56:18.367
So there's a group of data centers, there's another group of data centers and there's a third group of data centers and they're positioned very close to each other, but not too far, like very close, high latency, high bandwidth connection.

00:56:18.367 --> 00:56:25.059
Close, high latency, high bandwidth connection.

00:56:25.059 --> 00:56:29.927
And what happens is our databases, our paid production environments, are replicated instantly across these availability zones.

00:56:29.927 --> 00:56:57.483
So if the entire availability zone goes offline due to some issues like it's very rare when it happens, but when it happens the entire availability zone goes offline Our database in another availability zone becomes the primary one and the traffic is redirected to it without us doing anything on our end and without customers doing anything on their end.

00:56:57.483 --> 00:57:07.824
So even when the entire data center becomes unavailable, business central environment will remain online from another Azure availability zone.

00:57:07.824 --> 00:57:16.434
So that was another big feature we built for the online service to make sure that databases never go offline for these reasons.

00:57:16.434 --> 00:57:24.882
There may be many other reasons for databases to experience issues, but that will not be the issue.

00:57:26.014 --> 00:57:27.362
And that's included with their license.

00:57:28.056 --> 00:57:30.318
And, by the way, it's included with the license.

00:57:30.318 --> 00:57:33.817
So if you have paid licenses, paid production environment.

00:57:33.817 --> 00:57:35.724
That is a part of the offering.

00:57:37.659 --> 00:57:38.884
These are the subtle things.

00:57:38.884 --> 00:57:40.057
I call them subtle.

00:57:40.057 --> 00:57:51.557
They're significant but often overlooked when people look to see which and I'm not trying to sound like a sales pitch I'm just trying to make sure that individuals have the best implementation they can have and take many things into consideration.

00:57:51.557 --> 00:57:54.103
As you had mentioned, the Azure regions may go.

00:57:54.103 --> 00:57:57.858
It's rare that they go down, but it is a possibility.

00:57:57.858 --> 00:57:59.583
You could have a natural disaster.

00:57:59.583 --> 00:58:03.771
You could have something occur in an area where it does go offline.

00:58:03.771 --> 00:58:13.650
So knowing that there's already measures in place, that your business continuity is still in place is important, because that's you know.

00:58:13.650 --> 00:58:17.248
You put a server in your back office, for example.

00:58:17.248 --> 00:58:19.737
You know what happens if you have a natural disaster in the back.

00:58:19.737 --> 00:58:22.003
You know your back office the water pipe breaks.

00:58:22.664 --> 00:58:25.800
I've gone through that too, by the way, you know, my server gets, you know, get all wet.

00:58:25.800 --> 00:58:27.505
It's underwater because the water pipe broke.

00:58:27.505 --> 00:58:31.998
You know what do we do and you know, and this is the.

00:58:31.998 --> 00:58:51.516
These are some of the benefits that I think, if they were highlighted more as as you, would be well received uh so I'm happy to hear about that because, um, it's a, it's a good selling point, yeah, um by the way, our sla is is three nines right now.

00:58:51.615 --> 00:59:04.362
So nine 99.9 percent availability and in the observable past across the business central service we have not gone under this SLA.

00:59:04.362 --> 00:59:15.987
So for each particular environment it may have challenges, but the online service in total has never gone under 99.9 SLA.

00:59:15.987 --> 00:59:36.128
You know, that's also something we keep very, very close eye on and whenever anything happens which affects availability of an environment, that's a major investigation, that's a major, you know activity to understand what are the repair items to put into development and make sure it never happens again.

00:59:37.009 --> 00:59:38.210
Yep, it's amazing.

00:59:38.715 --> 00:59:39.896
No, there's a lot to it, and you also work with partner tooling.

00:59:39.896 --> 00:59:40.320
I recall, yep, it's amazing.

00:59:40.320 --> 00:59:43.681
No, there's a lot to it, and you also work with partner tooling.

00:59:43.681 --> 00:59:48.925
I recall, correct, yep, yes, so what does that encompass with your group?

00:59:49.856 --> 01:00:07.561
So that's a fantastic team of individuals working with AL language and Visual Studio Code and GitHub and GitHub so some of the top-of-the-line developer tools and team working with AppSource and all of our ISVs.

01:00:07.561 --> 01:00:25.643
So it is, by the way, we kind of stopped talking about this number because we talked about it too much, but we have, I think, much more than 5,000 apps on AppSource right now ISV apps and they are not dead apps.

01:00:25.643 --> 01:00:41.440
These are ISV solutions and some reseller solutions live on AppSource, installed, maintained, managed and covering pretty much every industry segment, pretty much every niche.

01:00:41.440 --> 01:00:43.706
We have apps for anything.

01:00:43.706 --> 01:00:46.519
In Business Central it seems like, oh yeah, there's a lot of apps.

01:00:46.780 --> 01:00:47.342
A lot of apps.

01:00:48.478 --> 01:00:57.065
So that is the team which is building the language, building the compiler, building the Visual Studio Code extension and maintaining the DevOps.

01:00:57.065 --> 01:01:03.804
You may have heard about our solution for DevOps ALGo built on GitHub.

01:01:03.804 --> 01:01:09.925
So the tooling, the set of tooling to help you manage your code as well in GitHub.

01:01:09.925 --> 01:01:13.262
So that's the development tools experience.

01:01:13.764 --> 01:01:14.666
No, it's a great tools.

01:01:14.666 --> 01:01:16.900
I'm very familiar with them, I use them.

01:01:16.900 --> 01:01:18.204
I use all of them.

01:01:18.514 --> 01:01:19.541
And millions of developers around the world would also be agreeing with you.

01:01:19.541 --> 01:01:20.072
No, it's a great set of tools, familiar with them, I used them.

01:01:20.072 --> 01:01:22.047
I use all of them, and millions of developers around the world would also be agreeing with you.

01:01:22.047 --> 01:01:23.248
No, no it's.

01:01:23.369 --> 01:01:24.409
It's a great set of tools.

01:01:24.409 --> 01:01:26.476
I remember app source back when it first started.

01:01:26.476 --> 01:01:28.802
I think you had it was all manually done.

01:01:28.802 --> 01:01:35.806
I think I remember the I forget the individual's name that I used to talk with when you put it in and he had to do everything manually uh early on.

01:01:35.806 --> 01:01:37.860
This is very early on many many ago.

01:01:37.860 --> 01:01:40.467
And now times have changed.

01:01:40.467 --> 01:01:42.442
You can submit it.

01:01:42.442 --> 01:01:45.456
It automatically goes through the entire process Once it's been in there.

01:01:45.456 --> 01:01:46.842
If you do an update, the validation.

01:01:46.842 --> 01:01:51.067
It's a process that I'm thankful is continuously improving.

01:01:51.795 --> 01:01:52.336
Like anything.

01:01:52.858 --> 01:01:56.788
It's time and use will drive change to improve the experience.

01:01:56.788 --> 01:02:02.666
At least we start out and we have that and you can have a number of applications in there.

01:02:03.396 --> 01:02:04.661
Wow my mind is growing.

01:02:06.018 --> 01:02:11.581
This automation, by the way, is also one of these wonders, similar to Point-in-Time Restore.

01:02:11.581 --> 01:02:23.967
Imagine that you have so many customers to support, so your solution is deployed across hundreds of customers or hundreds of customers, and then you need to deploy a hotfix or change to this environment.

01:02:23.967 --> 01:02:27.121
How do you go about this and anything else then?

01:02:27.121 --> 01:02:29.268
Then you know online environment.

01:02:29.268 --> 01:02:37.157
So with this, you submit your change, it gets processed, it gets validated across all of these country deployments.

01:02:37.157 --> 01:02:45.885
Business Central is Business Central has some 25 plus localizations, so your change gets validated against these localizations.

01:02:45.885 --> 01:02:48.103
Make sure that it compiles against all of them.

01:02:48.103 --> 01:02:51.914
It gets uploaded and available for all of your customers.

01:02:52.416 --> 01:02:57.349
The customers can go into their admin center and see that there is an update from ICD app.

01:02:57.349 --> 01:02:59.523
Click a button and get it installed.

01:02:59.523 --> 01:03:03.481
I mean within a matter of an hour.

01:03:03.481 --> 01:03:06.588
You can roll out your update if necessary.

01:03:06.588 --> 01:03:10.601
You can go as fast as you want, you can go as slow as you want.

01:03:10.601 --> 01:03:17.608
Your customers can decide to update, to do this update a week later, a month later, wherever they choose.

01:03:17.608 --> 01:03:27.184
But it is incredible the flexibility and the easiness of deploying updates into all of these environments by ICs, not just by Microsoft.

01:03:28.760 --> 01:03:37.264
I think it's a wonderful way of deployment because, as you mentioned, you just update the app, you say, okay, reinstall it, and they have the change.

01:03:37.264 --> 01:03:38.842
You don't have to send anybody anything.

01:03:38.842 --> 01:03:41.985
You don't have to get on the phone and walk them.

01:03:41.985 --> 01:03:43.731
Walk them through an install or an update.

01:03:43.731 --> 01:03:45.038
You know they can do it themselves.

01:03:45.038 --> 01:03:52.760
I mean, you may have to talk with them about the changes or, you know, I'm just saying for the implementation of the installation process, you may have another reason that you'd have to speak with them for it.

01:03:52.760 --> 01:03:54.905
But, uh, I don't know.

01:03:54.905 --> 01:03:55.487
It's been my life.

01:03:55.487 --> 01:04:01.856
So maybe maybe I, so maybe maybe I've.

01:04:01.856 --> 01:04:02.637
You know, I've been in it for too long.

01:04:02.637 --> 01:04:09.697
I can't think of, uh, any other ways, but I try to look at it, uh, you know, from outside eyes at times, just to, to, you know, make sure that I'm not missing anything.

01:04:09.697 --> 01:04:17.530
So, yeah, uh, well, well, dimitri, you, you filled my mind with a lot of, uh, great things.

01:04:17.530 --> 01:04:25.242
Uh, a couple new great links BC Security, bc Service Are those on the AKMS, bc All page as well.

01:04:25.362 --> 01:04:27.126
They should be Like if they're not?

01:04:27.126 --> 01:04:30.255
If they're not, we'll have to make a note to get them added.

01:04:30.416 --> 01:04:32.463
But I definitely watch out.

01:04:32.463 --> 01:04:38.014
I'll be posting those links later today, chris, don't do it before me, but I'll be sharing those.

01:04:38.215 --> 01:04:49.742
Those are two great links to share and I think it's important for partners and customers to be aware of them and you can add akamsbcperformance for sure to it, like that's a very well-known resource.

01:04:49.742 --> 01:04:54.898
This is where we describe guidance, not just for technical specialists but for the end users.

01:04:54.898 --> 01:05:02.628
I mean, use the latest version of your browser or use you know faster machine or things of that nature also help.

01:05:02.628 --> 01:05:03.469
Yes.

01:05:04.335 --> 01:05:05.822
Yeah, I'm just writing all these down.

01:05:05.822 --> 01:05:10.525
I'm actually I'm texting yeah, all of it is already written in that article DC performance.

01:05:10.715 --> 01:05:12.119
There's lots of great guidance there.

01:05:12.119 --> 01:05:13.644
Yeah, no, it's great For sure.

01:05:14.516 --> 01:05:16.342
Thank you again for taking the time to speak with us.

01:05:16.342 --> 01:05:19.538
We know that you have a busy schedule, you taking the time to speak with us.

01:05:19.538 --> 01:05:20.378
We know that you have a busy schedule.

01:05:20.378 --> 01:05:28.585
You must have a lot going on after coming off a vacation and you know, hearing all of the things that your amazing group is involved in and does, I'm sure you have a full schedule.

01:05:28.585 --> 01:05:31.068
So we appreciate you taking the time to speak with us.

01:05:32.007 --> 01:05:32.628
As I always say.

01:05:32.708 --> 01:05:34.590
I appreciate anybody spending time with me.

01:05:34.590 --> 01:05:36.472
Time is the currency of life.

01:05:36.472 --> 01:05:42.157
Once you spend it, you can't give it back.

01:05:42.157 --> 01:05:44.244
So anybody who gives me their time I'm grateful for and thank you again.

01:05:44.244 --> 01:05:45.289
Hope you have a great weekend.

01:05:45.309 --> 01:05:45.891
You get caught up, Thanks.

01:05:45.911 --> 01:05:46.251
Dimitri.

01:05:46.355 --> 01:06:03.135
And I look forward to Thanks a lot for inviting me, and I just wanted to mention in the end is that I, first of all, I do feel privileged talking to you, gentlemen, and talking to your listeners, so you're doing an amazing job, so that's very appreciated.

01:06:03.135 --> 01:06:15.451
We do want to spread this message that BC is evolving year after year to what you heard today and many other incredible things in the space of AI and other areas.

01:06:15.451 --> 01:06:24.610
But one thing which I wanted to share is the Business Central team keeps me amazed every day.

01:06:24.610 --> 01:06:31.909
So the team is an incredible set of individuals.

01:06:31.909 --> 01:06:37.985
They're so committed to the product when things happen in LifeSite and we try to stay humble.

01:06:38.155 --> 01:07:27.246
Yes, I may have mentioned a few incredible things that we do with the Business Central today, but things happen in the daily life and operation of the service which we want to improve, and every individual keeps accountable and bought in into the idea of making it happen, making it a success, and bought in into the idea of making it happen, making it a success, and that drive that's taking the responsibility, having no issues which are somebody else's issues, like everything is our own, everything is my own, like this mindset, this culture in the team, is such an incredible thing seem to see that I wanted to also send kudos to this unbelievable team of developers that are working on Business Central.

01:07:28.255 --> 01:07:34.175
And, by the way, it's the same vibe I'm getting from all partners engaged with Business Central.

01:07:34.175 --> 01:07:45.385
Every time you meet people at the conferences, in online forums, on Yammer, on our social media, there's very few people just chagging along.

01:07:45.385 --> 01:07:56.699
Everyone is chipped in and active and bought in and passionate, and the discussions we're having in this forum they're so passionate, they're so lively that they're not dry.

01:07:56.699 --> 01:08:10.684
They're never dry, and that's what keeps our team and me personally going that engaged partner community budding passionate individuals and this engaged team behind me in this building.

01:08:10.684 --> 01:08:12.545
It's an incredible setup.

01:08:12.545 --> 01:08:25.543
So I just wanted to say thank you to you for giving me this opportunity, thank you to all of our partners and thank you to the team building this amazing product and we see that we see that passion everywhere.

01:08:26.015 --> 01:08:29.002
Yes, it is this business central community.

01:08:29.002 --> 01:08:29.806
We do thank your team.

01:08:29.806 --> 01:08:31.193
We do recognize you have a team with everyone.

01:08:31.193 --> 01:08:43.820
We've spoken with the other product managers I forget the names, you guys, names change with Microsoft all the time Product group owners, product managers you know it's more than just you know the ones we get to speak with directly.

01:08:43.820 --> 01:08:48.222
You do have a great team behind you, because it does take many individuals to do all of these tasks.

01:08:48.222 --> 01:08:53.310
And one thing I can say about the business central community as well it is a passionate group.

01:08:53.310 --> 01:08:57.579
There's a lot of discussions that may be not always positive, but at least when I read them.

01:08:57.899 --> 01:09:02.608
it's out of passion, it's not anybody being, you know, malicious or anybody being hurtful.

01:09:02.608 --> 01:09:07.127
They're doing it because they're passionate and they enjoy and appreciate what they do.

01:09:07.127 --> 01:09:10.460
I'm thankful to be part of it, and it's changed over the years too, just like anything else.

01:09:10.460 --> 01:09:14.997
I remember when I first got into it you know back in late nineties.

01:09:15.439 --> 01:09:18.243
Um, you know as it was when it first came to the United States.

01:09:18.243 --> 01:09:19.706
The partner structure was so different.

01:09:19.706 --> 01:09:22.569
Now it is a true community.

01:09:22.569 --> 01:09:26.065
A lot of partners now have realized it's beneficial to help each other.

01:09:26.065 --> 01:09:30.703
You had mentioned there's so many updates to the products and I'll say every time how do you keep up?

01:09:30.703 --> 01:09:37.083
I have a hard time keeping up, and how you can keep up is you just need to have a good group of people that you work with or that you're associated with.

01:09:37.083 --> 01:09:43.167
That can help keep everybody moving forward and work with such a great product.

01:09:43.167 --> 01:09:48.850
Thank you again for your words and also give thanks to the team, tell, yannick and Vincent.

01:09:48.911 --> 01:09:52.858
I didn't forget Tell them when you talk with them afterwards, tell them we had a conversation.

01:09:52.858 --> 01:09:54.662
I didn't forget.

01:09:54.662 --> 01:09:57.479
I'll send you the email afterwards too, and they'll know what I'm talking about.

01:09:57.980 --> 01:09:59.345
Yes, thank you so much.

01:09:59.515 --> 01:10:00.436
But thank you again.

01:10:00.436 --> 01:10:08.384
We appreciate it and we look forward to talk with you soon and seeing you at some of the upcoming conferences as well my pleasure, all right, thank you ciao.

01:10:08.404 --> 01:10:10.376
Thank you see you take care, see you.

01:10:11.578 --> 01:10:20.485
Thank you, chris, for your time for another episode of in the dynamics corner chair, and thank you to our guests for participating thank you you, brad, for your time.

01:10:20.715 --> 01:10:23.944
It is a wonderful episode of Dynamics Corner Chair.

01:10:23.944 --> 01:10:27.445
I would also like to thank our guests for joining us.

01:10:27.445 --> 01:10:30.484
Thank you for all of our listeners tuning in as well.

01:10:30.484 --> 01:10:44.975
You can find Brad at developerlifecom, that is D-V-L-P-R-L-I-F-E dot com, and you can interact with them via Twitter D-V-L-P-R-L-I-F-E dot com, and you can interact with them via Twitter D-V-L-P-R-L-I-F-E.

01:10:44.975 --> 01:10:58.367
You can also find me at Mattalinoio, m-a-t-a-l-i-n-o dot I-O, and my Twitter handle is Mattalino16.

01:10:58.367 --> 01:11:02.063
And you can see those links down below in the show notes.

01:11:02.063 --> 01:11:03.407
Again, thank you everyone.

01:11:03.407 --> 01:11:04.980
Thank you and take care.