Oracle Server Outage : NOW SOLVED..

( Back-end server at Oracle Mumbai Data farm : SOLVED )

First date and time of reporting : 17-Sep-2023 : 00.30 am just past midnight

Current Status, : 17-Sep-2023 : 11.12 am : All complete. Mission Accomplished

There is a reported interruption in the Oracle Datafarm in India

Some Users facing problem of

TNS Listner not responding

Database not opening

etc

there is a outage ( disruption )
in Oracle servers

Some servers are working fine :  Restoration of the rest is in progress

Oracle Has informed that work is on to repair the problem, and hope to get back the things in control in next few minutes ( hopefully )

[01:03 am, 17/09/2023] Sangeet Kr Gupta: I spoke to Mr Ankur and Mr Mudit. Both are senior manager there, they confirmed the problem is at the Network Level at Mumbai Datafarm, and the Technical Team at Mumbai are on the job, to solve it

---

 

---

01.21 AM : ORACLE HAS UPDATED ITS OWN WEBSITE WITH THIS ALERT

 

Multiple Oracle Cloud Infrastructure (OCI) Services| India West (Mumbai) | 176dc5ab

---

---

( Update at 01.32 am )

Update at 01.32 am  IST : SKG Finsys HO, India Time. Problem deciphered at Airtel Data Connections at Mumbai , last mile connection as per Mr Mudit, at Oracle Head Office. Work in progress. He said, almost 250 servers are affected. Including the SaaS customers. It is a Big Outage, and all are on the job to rectify that.

---

( Update at 01.43 am )

Finsys HO, India Time. … As per Oracle HO, some servers restored. Call Domain being shifted to the “working servers” for the Finsys customers who are facing problem at night and are “ON” at this time.

 

( Update at 01.56 am )

Finsys HO, India Time. … As per Oracle HO,  more servers restored.

 

( Update at 02.05 am )

[1:54 am, 17/09/2023] Sangeet Kr Gupta: GOOD NEWS,

[1:54 am, 17/09/2023] Sangeet Kr Gupta: 2 out of our 17 servers “VM” are “ON” now

[1:54 am, 17/09/2023] Sangeet Kr Gupta: work in progress

[1:55 am, 17/09/2023] Sangeet Kr Gupta: will keep updating

[1:55 am, 17/09/2023] +91 84481 xxx86: Ok sir keep on updating

[1:56 am, 17/09/2023] +91 84481 xxx86: Vijay Gupta from xxxxxx gr noida

[2:01 am, 17/09/2023] Sangeet Kr Gupta: work in progress

[2:02 am, 17/09/2023] +91 84481 xxx86: Thanks for updating in group

( Update at 02.23 am )

[new server being created at Oracle Hyderabad by Finsys for our customers] But facing problem, since data from mumbai to be sourced via same network.

( Update at 03.09 am )

[3:04 am, 17/09/2023] Puneet Gupta Ji: make manual document
for preventing urgent supply problem
[3:04 am, 17/09/2023] Puneet Gupta Ji:  Use Govt’s e way bill portal in this emergency

( Update at 03.20 am )

The Situation is not normal. The server access at Mumbai is not yet proper. 

Request to Customers.

Please Use Govt’s e way bill portal in this emergency + Govt E invoice portal , + make invoice / Challan in Excel / word.

 

( Update at 03.31 am )

Even HDFC Bank Net Banking is Down… It is also on the same Datafarm. Problem is wider than expected.  …

 

( Update at 04.04 am )

We have created successfully a new server at Oracle Hyderabad ( their south India data farm). And now trying to pull data from Mumbai to Hyd.

( Update at 04.13 am )

{{ Mumbai Oracle Server Farm is still “DOWN” ,

So, this is a “emergency”. Please proceed to work on Excel / Word invoice for SOS supply as of now…  till things come back to normalcy }}

( Update at 04.45 am )

Oracle has asked for another 1-2 hours for problem to be resolved. Situation is not correct.

( Update at 05.01 am )

Oracle has asked for another 1-2 hours for problem to be resolved. Situation is not correct.  Finsys has requested Oracle to align with some specialist who can migrate data from Mumbai to Hyderabad Data centre… // restart from the old backups.

for time being

Please make E way bill and E invoices on the Govt Portal directly.. https://einvoice1.gst.gov.in/

 

( Good Update at 08.45 am )

[8:48 am, 17/09/2023] Abhay Gupta: Sirs, services are in process of resumption. Two largest servers are ok
[8:48 am, 17/09/2023] Abhay Gupta: Servers are starting back one by one

( Good Update at 08.45 am )

[9:01 am, 17/09/2023] Sangeet Kr Gupta: Good news is that many servers are getting restored back , 1 by 1
[9:02 am, 17/09/2023] Sangeet Kr Gupta: some of biggest customer servers have come back to normal functioning in last 10 minutes

( Good Update at 09.33 am )

4 servers are BACK normal working  …., and some more

Others… “work in progress”

 

( Good Update at 09.46 am )

Current Status, : 17-Sep-2023 : 09.48 am : Situation is coming back to normal… 9 out of 17 Finsys servers are 100% ok now. Rest are in waiting

[9:44 am, 17/09/2023] Sangeet Kr Gupta: xxxxx= DONE. OK
[9:44 am, 17/09/2023] Sangeet Kr Gupta: xxxxx = DONE
[9:44 am, 17/09/2023] Sangeet Kr Gupta: xxxxx = OK
[9:44 am, 17/09/2023] Sangeet Kr Gupta: xxxxx = OK DONE
[9:44 am, 17/09/2023] Sangeet Kr Gupta: xxxxx = OK DONE
[9:45 am, 17/09/2023] Sangeet Kr Gupta: xxxxx = OK DONE

( Good Update at 10.02 am )

[9:56 am, 17/09/2023] Abhay Gupta: 5 MORE COMPANIES  Server is working now !!! 

[10:03 am, 17/09/2023] Sangeet Kumar Gupta: good

[10:59 am, 17/09/2023] Sangeet Kumar Gupta: Mission complete
[11:00 am, 17/09/2023] Sangeet Kumar Gupta: @Adela Abhay Bansal ji, Pls confirm
[11:00 am, 17/09/2023] Adela Abhay Bansal: Thank you for your quick response, Sangeet ji

( Good Update at 10.14 am )

 

Current status at Oracle end

https://ocistatus.oraclecloud.com/#/incidents/ocid1.oraclecloudincident.oc1.phx.amaaaaaavwew44aacqf6p76wwqjx3dzdwibxhnbcnqgjx7zysrtdprn2jhua

22 out of about 55 Finsys clients on Oracle server are 100% ok now.

Finsys team members are updating the servers, Rebooting each from the Shell Level, as per the instruction of the Oracle Datafarm team, and one by one each server is coming back to life.

 

( Good Update at 10.37 am )

Current status at Oracle end = There site saying, 100% restored at network end. Asked us to restart servers and do some procedures. We are doing the same…. one by one most servers coming back to life.

Biomed, Acot, CHPL done at 10.38 am done

========

( Good Update at 11.02 am )

[11:01 am, 17/09/2023] Sangeet Kr Gupta: only 2 servers left
[11:01 am, 17/09/2023] Sangeet Kr Gupta: rest all done

Current Status, : 17-Sep-2023 : 11.04 am : Situation is coming back to normal… 43 out of about 51 Finsys clients on Oracle servers are 100% ok now. Rest are in waiting

========

( Good Update at 11.09 am )

Riki  server is ok now

CDS also

all done, all complete

Topic Closed at 11.12 am

Learnings (a) Proper DR = Disaster Recovery Systems must be in Place (b) Proper Backup must be in place (c) Midnight Infra must be in place.

 

MSME companies : ( defined as those will “non-critical” work, and downtime is acceptable) … Nothing to worry. No change required. Since such problems occur rarely.. maybe once in 1-2 years.

Mission Critical companies :

( defined as those will “Critical” work, and downtime is NOT acceptable) … Fast Decisions required.

Finsys can help you but via specialised agencies

Need to tieup with Service Providers who are specialising in these infra things like https://www.allatone.in/, and https://www.buchanan.com/, and https://www.mismosystems.com/  &  http://www.cybernoor.com

Hence, CAPA must be done as above.

CAPA, corrective action preventive action