Codesion Status Reports

ScrumWorks Pro Outage ( Fixed )

Date/Time: 2012-05-10 08:45 UTC

Outage Type: Unscheduled

Services Affected: ScrumWorks

Description: We currently experience problems with scrumworks server and we are working on to fix the issue. We will update this post once the issue is fixed.

Update: As of 10:45 UTC, all the scrumworks pro instances are up

Outage Billing page : Our Billing engine provider is currently having API calls problems.

Outage: Billing Engine
Date/Time: 2012-04-19 17:10 UTC
Affected Services: Codesion Billing Page, and new signups (No other services are affected)
Outage Type: FIXED
Duration: 2hrs

Description: Our Billing engine provider is currently having problems with their API calls, and they are actively working on resolving the issue.

Outage: api.codesion.com (Fixed)

Outage: Codesion API
Date/Time: 2012-04-11 19:50 - 23:00 UTC
Affected Services: Just Codesion API usage, all other services are working
Outage Type: Unscheduled
Description: We are looking into this issue. Update: api.codesion.com is up now.

Maintenance: Cold Backup Issues ( Resolved )

Date/Time: 2012/04/08 - 2012/04/09

Outage Type: Unscheduled

Services Affected: Cold Backups

Description: We found a problem with our backup servers. Due to this backups might not have pushed to your destinations. You might also have noticed email notifications for ondemand backups are not sent to you. Problem is fixed and the backup server is working properly now.

Maintenance: Software Upgrade Various Server 2012-04-09 (Completed)

Date/Time: Mon Apr 09, 2012 07:30 - 10:30 UTC

Outage Type: Scheduled( Immediate )

Servers: CVS01, Push01(Publisher), Bugz01, Git02

Description: We will be doing software upgrade on above mentioned window. Software upgrade requries reboot of the server which will cause short outage.

Update: CVS01, Bugz01 and Git02 are upgraded. Push01 will be scheduled to a later date.

Outage: SWP 2012-04-05 (Fixed)

Outage: ScrumWorks Pro
Date/Time: 2012-04-05 10:30-12:30 UTC
Affected Services: ScrumWorks Pro
Outage Type: Unscheduled
Duration: Two Hours

Description: Our ScrumWorks Pro system experienced some problems during this time period.

Announcement: TeamForge Association Viewer

TeamForge Association Viewer is now availabe.

This addon displays the associations/dependancices graphically.

To use the Addon navigate to an Artifact -> Associations/Dependancies -> Visualize

Trac-Subversion Link Problems 2012-04-05

Outage: Trac-Subversion Link Problems
Date/Time: 2012-04-05 01:00-02:20 UTC
Affected Services: Trac06 & SVN07
Outage Type: Unscheduled
Duration: Approximately one hour and twenty minutes

Description: There was a problem with the link between these two servers which is used for the Trac to Subversion integration. This resulted in the Ticket commit integrations failing, and Trac's Subversion browser failing with an error similar to this: "Warning: Can't synchronize with repository "(default)" (/services/svn07/svn/ORG/PROJ does not appear to be a Subversion repository.). Look in the Trac log for more information."

This issue is resolved now, and we are performing some preventative maintenance to assure that it does not reoccur.

Maintenance: DB01 (Completed, with impacts to all applications)

Date/Time: Saturday, April 7, 2012 2:30 AM to 5:30 AM17:30 to 20:30 UTC
Servers Affected: DB01 (and ALL CODESION SERVICES)
Outage Type: Scheduled ( Immediate )
Duration: Actual downtime is expected to be around 30 minutes during this window, but ALL CODESION SERVICES may be affected. Total downtime may not be one service outage during this time window, but several shorter ones during the three hour period.

Update: DB01 was down for 40 minutes (which means most of our services were also down for that time). Sadly, we also had a problem with Publisher, which was resolved after 2 hours after the window. Also, we had the incorrect time above for the outage window due to timezone conversion errors. It was scheduled for 17:30 UTC on Saturday.

Maintenance: HOT06

Date/Time: Thursday, April 5th, 2012, 6:00am - 9:00am (UTC)

Servers Affected: HOT06

Outage Type: Scheduled ( Immediate )

Description: We will be doing hardware maintenance for hot06. Although the maintenance window is set for 3hrs, we don't expect more than 30 minutes of downtime.

No customer facing services will be affected during this maintenance window.